Real-time system for multi-modal 3D geospatial mapping, object recognition, scene annotation and analytics

ABSTRACT

A multi-sensor, multi-modal data collection, analysis, recognition, and visualization platform can be embodied in a navigation capable vehicle. The platform provides an automated tool that can integrate multi-modal sensor data including two-dimensional image data, three-dimensional image data, and motion, location, or orientation data, and create a visual representation of the integrated sensor data, in a live operational environment. An illustrative platform architecture incorporates modular domain-specific business analytics “plug ins” to provide real-time annotation of the visual representation with domain-specific markups.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. ProvisionalApplication Ser. No. 61/954,635, filed Mar. 18, 2014, and U.S.Provisional Application Ser. No. 62/074,674, filed Nov. 4, 2014, each ofwhich is incorporated herein by this reference in its entirety.

GOVERNMENT RIGHTS

This invention was made in part with government support under contractnumber N00014-12-C-0070 awarded by the Office of NavalResearch-Arlington. The Government has certain rights in this invention.

BACKGROUND

This disclosure relates to the technical fields of computer vision,mobile robot navigation, and geospatial mapping and analysis. Incomputer vision, mathematical techniques are used to detect the presenceof and recognize various elements of the visual scenes that are depictedin digital images. Localized portions of an image, on which specifictypes of computations are done to produce visual features, may be usedto analyze and classify the image. Low-level and mid-level features,such as interest points and edges, edge distributions, colordistributions, shapes and shape distributions, may be computed from animage and used to detect, for example, people, objects, and landmarksthat are depicted in the image. Machine learning algorithms are oftenused for image recognition.

In robot navigation technology, cameras and other sensors are used todetermine the robot's location and orientation with respect to itssurrounding real world environment (i.e., the robot's frame ofreference). Computer vision techniques and mathematical computations areperformed to interpret digital images of the environment within therobot's frame of reference, generate a mathematical representation ofthe environment, and generate a mapping of objects in the real world tothe mathematical representation of the environment (e.g., a “map”). Therobot uses the map to navigate about its environment. In order tonavigate, the robot performs mathematical computations to develop anavigational path to a goal location.

Geospatial technology relates to the acquisition, analysis, andpresentation of geographical and/or geospatial data, such as GlobalPositioning System (GPS) data and geographic information system (GIS)data.

BRIEF DESCRIPTION OF THE DRAWINGS

This disclosure is illustrated by way of example and not by way oflimitation in the accompanying figures. The figures may, alone or incombination, illustrate one or more embodiments of the disclosure.Elements illustrated in the figures are not necessarily drawn to scale.Reference labels may be repeated among the figures to indicatecorresponding or analogous elements.

FIG. 1 is a simplified schematic diagram of an environment of at leastone embodiment of a computing system including a multi-sensor datacollection, analysis, recognition, and visualization platform asdisclosed herein;

FIG. 2 is a simplified schematic diagram of an environment of at leastone embodiment of the multi-modal navigation and geo-spatial mappingsubsystem of FIG. 1, including a 6DOF pose estimation module asdisclosed herein;

FIG. 3 is a simplified schematic diagram of an environment of at leastone embodiment of the multi-modal live analytics subsystem of FIG. 1,including an object/scene recognition and classification subsystem asdisclosed herein;

FIG. 4 is a simplified schematic diagram of an environment of at leastone embodiment of the real-time multi-modal compression subsystem ofFIG. 1, including a multi-level compression module as disclosed herein;

FIG. 5 is a simplified schematic diagram of an environment of at leastone embodiment of the multi-platform data aggregation subsystem of FIG.1;

FIG. 6 is a simplified schematic diagram of an environment of at leastone embodiment of the 6DOF pose estimation module of FIG. 2;

FIG. 7A is a simplified schematic diagram of an environment of at leastone embodiment of the object/scene recognition and classificationsubsystem of FIG. 3;

FIG. 7B is a simplified schematic diagram of an illustrative layeredobject recognition architecture of at least one embodiment of theobject/scene recognition and classification subsystem of FIG. 3;

FIG. 8 is a simplified schematic diagram illustrating an example oflarge scale object recognition and scene classification of at least oneembodiment of the object/scene recognition and classification subsystemof FIG. 3;

FIG. 9 is a simplified schematic diagram illustrating an example ofcontext-free object identification with feature sharing, of at least oneembodiment of the object/scene recognition and classification subsystemof FIG. 3;

FIG. 10 is a simplified schematic diagram illustrating an example ofcontextual object identification and classification, of at least oneembodiment of the object/scene recognition and classification subsystemof FIG. 3;

FIG. 11 is a simplified schematic diagram illustrating another exampleof contextual object identification and classification, of at least oneembodiment of the object/scene recognition and classification subsystemof FIG. 3;

FIG. 12 is a simplified example of output of an application of theobject/scene recognition and classification subsystem of FIG. 3 usingthree-dimensional aerial LIDAR data, including a set of annotateddigital images;

FIGS. 13A and 13B are simplified examples of output of an application ofthe change detection module of FIG. 3 using three-dimensional aerialLIDAR data, including an annotated digital image;

FIG. 14 is a simplified schematic diagram illustrating datarepresentation and compression operations that may be performed by themulti-level compression module of FIG. 4;

FIG. 15 is a simplified example of output of an application of themulti-level compression module of FIG. 4, including a set of digitalimages;

FIG. 16 is a simplified flow diagram of at least one embodiment of aprocess by which the computing system of FIG. 1 may perform collection,analysis, recognition, and visualization operations on multi-sensor,multi-modal data as disclosed herein;

FIG. 17 is a simplified flow diagram of at least one embodiment of aprocess by which the computing system of FIG. 1 may perform objectrecognition and classification using multi-sensor, multi-modal data asdisclosed herein;

FIG. 18 is a simplified schematic diagram of at least one embodiment ofa real time multi-processor computing system for performing sceneanalytics as disclosed herein; and

FIG. 19 is a simplified block diagram of an exemplary computingenvironment in connection with which at least one embodiment of thesystem of FIG. 1 may be implemented.

DETAILED DESCRIPTION OF THE DRAWINGS

While the concepts of the present disclosure are susceptible to variousmodifications and alternative forms, specific embodiments thereof areshown by way of example in the drawings and are described in detailbelow. It should be understood that there is no intent to limit theconcepts of the present disclosure to the particular forms disclosed. Onthe contrary, the intent is to cover all modifications, equivalents, andalternatives consistent with the present disclosure and the appendedclaims.

Mobile robots, vehicles, and other mobile computing devices can beequipped with a number of different types of “live” sensors, such ascameras, LIDAR, and GPS, which can collect a range of different types ofinformation about the real world physical environment surrounding thedevice, in real time. The technology disclosed herein can, in real time,integrate the diverse sensor data produced by these and/or othermultiple different types of live sensors, including three-dimensionaldata, in order to provide real-time analysis and mapping of geospatialareas using an automated tool. The disclosed technology can analyze andannotate the integrated sensor data using domain-specific business logicand/or more ‘generic’ analytics, as needed. For example, the disclosedarchitecture enables the use of “plug in” data analytics modulesdesigned for specific applications, such as surveillance, situationawareness, facility and equipment monitoring, asset monitoring, disasterrelief, search-and-rescue, and/or other applications. Integrating livemulti-sensor multi-modal data, including three-dimensional (3D) data, ina real-time system as disclosed herein can, among other things, enablelow-latency map updates and analytics to be delivered to requestingusers and/or services in real time or interactively. Additionally, theintegration of data produced by multi-modal sensors, such as cameras,LIDAR, and GPS, can provide improved map accuracy and enable a broadrange of analysis and information products.

Referring now to FIG. 1, an embodiment of a computing system 100implementing an automated tool for the collection, analysis, recognitionand visualization of multi-sensor multi-modal data, including 3D data,is shown. In FIG. 1, the computing system 100 is shown in the context ofan environment 101 that may be created during the operation of thecomputing system 100 (e.g., a physical and/or virtual execution or“runtime” environment). The computing system 100, and each of theplatforms, subsystems, modules, and other components of the computingsystem 100, is embodied as a number of machine-readable instructions,data structures and/or other components or devices, which may beimplemented as computer hardware, firmware, software, or a combinationthereof.

The illustrative computing system 100 includes a multi-sensor datacollection, analysis, recognition, and visualization platform(“platform”) 120. The illustrative platform 120 may be embodied as amobile computing device, including a navigation-capable device such as amobile robot, an unmanned or unpiloted ground and/or aerial vehicle(e.g., a drone, a “driverless” vehicle, etc.), a motorized vehicle, oreven in some cases as a personal mobile electronic device (e.g., smartglasses, smart appliance, etc.). In other embodiments, the platform 120is embodied as another type of computing device, such as a component ofa security or surveillance system that is mounted to a wall or fixture,or as a combination of multiple devices.

In operation, the platform 120 receives sensor data streams 112, 114,116 from a number of sensors 102, 104, 106. The sensors 102, 104, 106are of “N” different sensor types (where “N” is a positive integer). Forexample, the sensor 102 may be embodied as a two-dimensional (2D) imagesensor (e.g., a 2D still or video camera); the sensor 104 may beembodied as a 3D image sensor (e.g., LIDAR), and the sensor 106 may beembodied as an inertial sensor such as an inertial measurement unit(IMU) (including, e.g., an accelerometer and a gyroscope) or anothertype of sensor capable of producing motion, location, and/or orientationdata. In the illustrative embodiments, the sensor 106 is a “low-end”IMU, such as a Micro-Electro-Mechanical Systems (MEMS) IMU that isreadily commercially available and thus suitable for use in connectionwith a wide range of consumer-grade applications. In other embodiments,the sensors 102, 104, 106 may include a “higher-end” IMU and GPS-basedintegrated navigation system.

The illustrative platform 120 is embodied as a “software pipeline”including a set of interconnected modules including a sensor datacapture and synchronization module 122, a multi-modal navigation andgeo-spatial mapping subsystem 126, a multi-modal geo-spatial dataintegration module 130, a multi-modal live analytics subsystem 134, adomain-specific data correlation module 136, a real-time multi-modalcompression subsystem 142, and a live on-platform visualization module146. In other embodiments, any of the modules/subsystems 122, 126, 130,134, 136, 142, 146 may be embodied as hardware, firmware, software, or acombination thereof.

The modules/subsystems 122, 126, 130, 134, 136, 142, 146 interface withthe live sensors 102, 104, 106, align and analyze the sensor data 112,114, 116, correlate the sensor data 112, 114, 116 with domain-specificbusiness data (e.g., domain-specific data/maps 150 stored at leasttemporarily in a domain-specific data/map cache 156), compress andtransmit the resulting information (e.g., compressed multi-modal data144, which may include annotations such as geo-tags 138 and/or markups140, as described below), and provide visualization and interactive userinterfaces on the platform 120 and/or other computing devices, asdescribed in more detail below. Unlike other approaches, embodiments ofthe platform 120 can operate in real-time, and can combine multi-sensor,multi-modal navigation, mapping, analytics, compression, andvisualization functionality, all in a single architecture.

Platform Architecture

Referring further to FIG. 1, the illustrative sensor data capture andsynchronization module 122 operates as front-end software thatcommunicates with the sensors 102, 104, 106, synchronizes the sensordata streams 112, 114, 116, and timestamps the sensor data 112, 114, 116such that the data 112, 114, 116 from all of the sensors 102, 104, 106can be utilized concurrently (e.g., simultaneously). The sensor datacapture and synchronization module 122 outputs synchronized sensor data124 for use by the multi-modal navigation and geospatial mappingsubsystem 126. The illustrative multi-modal navigation and geospatialmapping subsystem 126 uses the synchronized sensor data 124 to maketemporal, spatial and geospatial associations across the multi-modalsensor data 112, 114, 116, in order to estimate both a navigation pathof the platform 120 and also to build up map data (e.g., aplatform-internal representation of the real world environmentsurrounding the platform 120, with respect to a frame of reference ofthe platform 120). The multi-modal navigation and geospatial mappingsubsystem 126 outputs multi-modal navigation and mapping data 128 foruse by the multi-modal geospatial data integration module 130. Themulti-modal geospatial data integration module 130 executes dataintegration and data fusion algorithms on the multi-modal navigationpath and mapping data 128 to produce integrated multi-modal geospatialdata 132, e.g., geo-spatially organized 3D maps. The integrated data 132is used by the multi-modal live analytics subsystem 134 and thereal-time multi-modal compression subsystem 142. The multi-modal liveanalytics subsystem 134 applies different object and/or scene detectionand recognition algorithms to the geo-spatially integrated data 132, andoutputs annotations of the integrated data 132, e.g., live analyticsgeo-tags 138. Whereas the analytics performed by the multi-modal liveanalytics subsystem 134 may utilize “generic” algorithms in someembodiments (e.g., standard vehicle or face detection algorithms), thedomain-specific data correlation module 136 applies domain-specificbusiness logic for a selected (e.g., specialized) field-of-interest, andoutputs domain-specific markups 140 for the integrated data 132, for useby the compression subsystem 142. The annotations produced by the liveanalytics subsystem 134 and/or the domain-specific data correlationmodule (e.g., the geo-tags 138 and the domain-specific markups 140) maytake the form of, for example, graphical overlays, such as color-codedhighlighting, graphical symbols, markings such as lines, circles andarrows, etc. In some embodiments, these annotations may be interactivesuch that if activated by a user (e.g., by a tap on a touchscreendisplay showing a visualization of the compressed multi-modal data 144),they may display additional information, provide an expanded user inputarea into which a user may input notes or other information, or useractivation of the interactive annotation may launch or otherwise provideaccess to another application or service, such as a messaging service.

The architecture of the platform 120 allows for each or either of themulti-modal live analytics subsystem 134 and the domain-specific datacorrelation module 136 to be implemented as “plug in” modules, in thatthe live analytics subsystem 134 and/or the data correlation module 136can be selected based on the needs or the design of a particularimplementation of the platform 120. The real-time multi-modalcompression subsystem 142 utilizes compression algorithms that develop a3D representation of the integrated data 132 that is suitable forcompression and live streaming, and outputs the compressed multi-modaldata 144 (e.g., annotated 3D-2D data streams) for use by the liveon-platform visualization module 146 and/or a multi-platform dataaggregation subsystem 148, which is illustratively located off of theplatform 120. The live on-platform visualization module 146 prepares adomain-specific visualization of the collected and integrated map data132 and analytics results (e.g., geo-tags 138, markups 140), which canbe accessed directly on the platform 120 in the field, or at anothercomputing device (e.g., a computing device of the multi-platform dataaggregation subsystem 148). The multi-platform data aggregationsubsystem 148 receives the compressed multi-modal data 144 from theplatform 120 and, in some embodiments, other platforms as well. The dataaggregation subsystem 148 fuses the annotated multi-modal data 144received from all platforms 120 into a centralized aggregated mapdatabase, which may be stored in a data storage device on the platform120 or on one or more other devices (e.g., data storage device 1918and/or 1978, shown in FIG. 19). The data aggregation subsystem 148applies business logic to the aggregated multi-modal data 144 tointerpret the aggregated data alone or in combination with otherbusiness-specific data that is accessible to the data aggregationsubsystem 148. The crowd-sourcing app 152 may be embodied as, forexample, a business-specific mobile application for modern smartphonesor other personal electronic devices. Data made available by thecrowd-sourcing app 152 can be used to augment the business analyticsoutput and 3D map information (e.g., integrated map data 132, geo-tags138, markups 140), in real-time. For instance, the crowd-sourcing app152 may supply domain-specific geo-tags 154 that have been obtained orderived from the public crowd-sourcing of information over the Internet.

As shown in FIG. 19, described below, the platform 120 may communicatewith one or more other computing devices, such as user computing devices(e.g., desktop, laptop, or tablet computers) and enterprise or “commandcenter” computing devices (e.g., servers, networks of servers, etc.).For example, the platform 120 may transmit data collected from thesensors 102, 104, 106 and processed by the platform 120 to themulti-platform data aggregation subsystem 148, which may be embodied ina data aggregation computing device 1970, and the crowd-sourcingapplication 152 may be installed on a user computing device 1950.

Data Synchronization

Referring now in more detail to the sensor data capture andsynchronization module 122 of FIG. 1, the illustrative sensor datacapture and synchronization module 122 reads the data 112, 114, 116output by the sensors 102, 104, 106 continuously while the platform 120is in operation. The data capture and synchronization module 122 tightlyintegrates and synchronizes the data 112, 114, 116 before publishing thesynchronized sensor data 124 for use by the navigation and mappingsubsystem 126 and the data integration module 130. Illustratively, thedata capture and synchronization module 122 represents each type ofsensor 102, 104, 106 (e.g., LIDAR, inertial measurement unit (IMU),camera) in software by an abstract class that can be instantiated usingdifferent drivers and which can be configured differently for differenttypes of sensor devices. The data capture and synchronization module 122reads the data 112, 114, 116 asynchronously or synchronously from eachsensor device 102, 104, 106, timestamps the data 112, 114, 116, andpublishes the data 124 for use by the downstream modules (e.g., modules126, 130, etc.). The processes performed by the data capture andsynchronization module 122 helps ensure low-latency of the pipeline,such that the delivery of geospatial analytics and mapping data occursin real time, or at least in interactive time.

Navigation and Mapping

Referring now to FIG. 2, components of an embodiment of the multi-modalnavigation and geospatial mapping subsystem 126 of FIG. 1 are shown inmore detail. Each of the components shown in FIG. 2 may be embodied assoftware, hardware, firmware, or a combination thereof. In FIG. 2, thecomponents of the navigation and geospatial mapping subsystem 126 areshown in the context of an environment 201 that may be created duringthe operation of the computing system 100 (e.g., a physical and/orvirtual execution or “runtime” environment). The illustrative navigationand mapping subsystem 126 includes a temporal association module 210, aspatial association module 212, a geospatial association module 214, anda multi-modal localization and mapping module 216 (which includes a 6DOFpose estimation module 218, described in more detail below withreference to FIG. 6). The components of the navigation and mappingsubsystem 126 operate to maintain accurate position and orientationinformation for each of the sensors 102, 104, 106, and make temporal,spatial and geospatial associations across the data 112, 114, 116obtained from the various different sensors 102, 104, 106. Theoperations performed by the components of the navigation and mappingsubsystem 126 allow the map representation 128 to be created over time.The navigation and mapping subsystem 126 is capable of exploitingvarious combinations of multi-modal observations derived from the data124 to generate a navigation path and map 128 for the platform 120. Forexample, the navigation and mapping subsystem 126 can utilize acombination of data obtained from an IMU and video data to estimate aninitial navigation path for the platform 120, and then use the relative6 degrees of freedom (6DOF) poses that are estimated by the 6DOF poseestimation module 218 to generate a 3D map based on data obtained from,e.g., a scanning LIDAR sensor. The 3D LIDAR features can then be furtherexploited to improve the navigation path and the map previouslygenerated based on, e.g., the IMU and video data.

Components of the navigation and mapping subsystem 126 computenavigation estimation for the platform 120. For reliable integration ofLIDAR and other multi-modal sensor data, accurate geospatial positionand orientation estimates of the sensor system (e.g., the combination ofsensors 102, 104, 106) are needed. Whereas other approaches rely onexpensive, high-end IMU and GPS-based integrated navigation systems toprovide the requisite position and orientation estimates, the disclosednavigation and mapping subsystem 126 can operate equally as well using,e.g., a MEMs IMU-based system. As described below with reference to FIG.6, the navigation and mapping subsystem 126 integrates other sensormeasurements, such as visual features obtained from 2D and/or 3D imagingdevices, with the position and orientation information, to enable robust6DOF motion estimation.

Data Integration

Referring now in more detail to the multi-modal geo-spatial dataintegration module 130 of FIG. 1, components of the illustrativemulti-modal geo-spatial data integration module 130 integrate and fusethe multi-modal navigation path and mapping data 128 in a 3D geospatialmap. The accurate position estimation of the different sensors 102, 104,106 with respect to real world coordinates and to the other sensors 102,104, 106, performed by the navigation and mapping subsystem 126,combined with the accurate synchronization of the sensor data 112, 114,116 provided by the data capture and synchronization module 122, allowsthe data integration module 130 to integrate and fuse the data 128 in a3D geospatial map. For example, a 3D map generated from LIDAR can beused to identify hyper-spectral or RGB video features associated withthe 3D surface generated from the map. Illustratively, the dataintegration module 130 fuses the sensor data 112, 114, 116 by combiningthe output of the different data sources (e.g., sensors 102, 104, 106)into a single channel or layer of the integrated map. For example, theoutput of different cameras with overlapping fields of view may becombined into a single RGB overlay, or the LIDAR points may be givencolor attributes from the overlay of the RGB or Hyper-Spectral imagery.The output of the data integration module 130 is geospatially alignedand integrated multi-modal data 132, which can be accessed for immediatecompression and visualization by the compression subsystem 142 andvisualization module 146, and/or analyzed by the analytics subsystem 134and data correlation module 136.

Live Analytics and Domain-Specific Data Correlation

Referring now to FIG. 3, components of an embodiment of the multi-modallive analytics subsystem 134 of FIG. 1 are shown in more detail. Each ofthe components shown in FIG. 3 may be embodied as software, hardware,firmware, or a combination thereof. In FIG. 3, the components of themulti-modal live analytics subsystem 134 are shown in the context of anenvironment 301 that may be created during the operation of thecomputing system 100 (e.g., a physical and/or virtual execution or“runtime” environment).The illustrative multi-modal live analyticssubsystem 134 includes a change detection module 310, an object/scenerecognition and classification subsystem 312, and an anomaly detectionmodule 314. The illustrative object/scene recognition and classificationsubsystem 312 is described in more detail below with reference to FIGS.7A and 7B. The components of the live analytics subsystem 134 computelive analytics on the 3D multi-modal data 132 to extract higher-levelinformation constructs that can be exploited by other modules and/orservices. The platform 120 provides a plug-and-play framework to plug inanalytics modules according to the needs of a particular application.The recognition and classification component 312 of the live analyticssubsystem 134 applies feature and attribute extraction methods to 2Dand/or 3D multi-modal data 132. Detection and classification tools ofthe recognition and classification component 312 allow for identifyingdifferent objects of interest (e.g. roads, buildings, poles, wires,trees, foliage, people, animals, fire hydrants, transformers,sign-posts, billboards, bridges, vehicles etc.).

A change detection module 310 can compare the current multi-modal map132 to a reference map or a previously-generated map, in order toidentify areas of the map that have changed since an earlier round ofdata collection. An anomaly detection module 314 can identify regionsthat do not fit the norm (e.g., obstructed roadways or damaged buildingsor tree limbs hanging over wires or poles down). The change detectionmodule 310 allows the platform 120 to identify “real” changes (e.g.,changes that are not due to occlusion) while reducing false alarms. Inthe illustrative system 100, the change detection module 310 exploitsboth 3D and 2D (e.g., video/hyperspectral) information for identifyingchanges in the data over time. In performing 3D change detection, thechange detection module 310 distinguishes differences in the data thatare due to occlusions from differences in the data that are due toactual change. The system 100 can be configured to (actively orpassively) ensure that sufficient looks are maintained to avoidocclusions. With the occlusions omitted or ignored, the basic 3D andimage changes can then be fed into a higher-level change evaluationmodule. In some embodiments, multi-modal training data is used to learnfeatures of robust change and for suppression of false alarms. Suchtraining is performed through, e.g., supervised training based on groundtruthing pre-collected data, or by online observation ofuser-in-the-loop nominations.

FIGS. 13A and 13B show examples of visualization output resulting fromthe operations of an embodiment of the live analytics subsystem 134,including operations of the change detection module 310. FIG. 13Aillustrates a visualization 1301 of a map of 3D and image data of aparticular geographic location, which is stored in a reference database.FIG. 13B illustrates a visualization 1302 of a map of 3D and image dataof the same geographic location as in FIG. 13A, taken at a later pointin time than the map of FIG. 13A. In the visualization of FIG. 13B,annotations 1314, 1316 indicate areas of the map that have changed incomparison to the prior map 1301. Notably, the changes detected by thelive analytics subsystem 134 and highlighted on the visualization 1302do not simply reflect changes due to occlusion or frame of reference.Rather, the annotations indicate actual changes in physical features ofthe real world environment depicted by the map 132.

In some embodiments, the general-purpose 3D analytics provided by thelive analytics subsystem 134 are, by the domain-specific datacorrelation module 136, further expanded within the plug-and-playframework of the platform 120 to bring in specific business logic forone or more specialized fields-of-interest. The “plug-and play” aspectof the platform 120 provides a facility for accessing the business dataand exploitation tools that is separate from and maintained outside themain sensor data streaming pipeline but which can utilize the sensordata. For example, a road-survey service can introduce specific logicabout materials used in a road-construction process, in order to improvethe detection of faults. Another example is a first responder team thatmight introduce a wide-area flood plain analysis module and buildinglocation and road-network map to determine real-time evacuation routes.A facility monitoring analysis module may detect fallen poles or foliageand tree branches touching wires, etc. A power company may use specificknowledge about the network of wire connections to determine the effectof a broken wire detected by the platform 120.

The system 100 provides a general-purpose geospatial caching frameworkto bring in the reference data 410 (e.g., prior data, business data)that includes information relevant to the current geospatial context.The reference data 410 is illustratively indexed spatially, using, e.g.,an octree data structure, allowing for fast queries of data within agiven area of interest. The data is automatically loaded from disk uponquery if it is not in memory, and pushed to disk when no longerrelevant. This type of general-purpose data caching framework allowslarge-scale storage and utilization of data and can accelerate theintegration of new data types and new business logic.

Data Representation and Compression

Referring now to FIGS. 4 and 14, components of an embodiment of thereal-time multi-modal compression subsystem 142 of FIG. 1 are shown inmore detail. Each of the components shown in FIGS. 4 and 14 may beembodied as software, hardware, firmware, or a combination thereof. InFIG. 4, the components of the multi-modal compression subsystem 142 areshown in the context of an environment 401 that may be created duringthe operation of the computing system 100 (e.g., a physical and/orvirtual execution or “runtime” environment). The illustrative real-timemulti-modal compression subsystem 142 includes data stores for referencedata updates 410, geo-tags 412, and domain-specific markups 414, as wellas a multi-level compression module 416. The reference data 410 includesupdated geospatial data 132 generated by the data integration module130. The geo-tags 412 are obtained as a result of analytics performed bythe live analytics subsystem 134. The domain-specific markups 414 areobtained as a result of the correlations performed by thedomain-specific data correlation module 136. In other words, theillustrative data stores storing reference data 410, geo-tags 412, andmarkups 414 contain only the relevant data updates and annotationsprepared based on the most recent set of sensor data 112, 114, 116,rather than the entire set of integrated multi-modal geospatial data132. Thus, the multi-level compression module 416 operates using onlythe updated/changed set of reference data 410 and annotations (e.g.,geo-tags 412, markups 414) as inputs to its compression algorithms.

The compression subsystem 142 enables efficient compression and robusttransmission of the data between the air/ground mobile computingplatform 120 and other devices, such as a “command center” computingsystem. The resulting data compression helps ensure real-time (or atleast interactive time) delivery of data and analytics results torequesting users and/or services. As shown in FIG. 14, some embodimentsof the compression subsystem 142 utilize an octree-forest basedrepresentation of map data, which can transmit the changes (e.g.,additions) to the 3D collection of data 410, geo-tags 412, and markups414 to a remote receiver (such as the data aggregation subsystem 148) asstreaming elements. Other sources (such as video/HSI) can exploit the 2Ddata layers to improve compression. The illustrative compressionsubsystem 142 also provides for the handling of dropped data packets andcorruption from faulty transmission in order to ensure resilientcommunication.

In some embodiments, the compression module 416 utilizes a sparselyoccupied yet densely mapped 3D voxel grid, supported by an octree forestdata structure. A summary of this technology is provided below. Furtherdetails are provided in the aforementioned U.S. Provisional ApplicationSer. No. 62/074,674, filed Nov. 4, 2014, which is entitled “StreamingPoint Cloud Visualization.” Voxel discretization size can be specifiedto be within the error threshold of the used depth sensing device (thusproviding semi-lossless representation), or a preferred size. Pointsfrom the sensor are globally integrated into a world coordinate frame.Points then fall within the spatial region of prospective voxels,causing those voxels to become occupied. A mirrored voxel grid ismaintained between the visualization “client” (e.g., a data aggregationplatform) and the “server” (e.g., a robot), and differential changes tothis grid are compressed and transmitted to the client, thus maintainingsynchronized occupancy representations.

The world model can be understood as a collection of occupied cellswithin an infinitely extendable 3D regular grid of a specifiedresolution (voxel size), thus discretizing 3D space into a lattice ofaligned addressable volumes. Grid cell occupancy is Boolean in nature,either being occupied or not. If at least one integrated point fallswithin a cell volume, that cell is considered occupied and anysubsequent points falling within the same volume are consideredredundant and ignored. An occupied cell is represented as a voxel andstored within the data representation. In computer-based modeling orgraphic simulation, a “voxel” may refer to each element of an array ofelements of volume that constitutes a notional three-dimensional space,e.g., each element of an array of discrete elements into which arepresentation of a three-dimensional object is divided.

In some embodiments, the data representation used to hold the worldvoxel occupancy information is an octree forest F that is defined to bea 3D array of forest cells C, each containing an octree O, arranged in aregular grid. In these embodiments, all octrees are of equal size andequal depth, where all leaves reside at the maximum depth of the tree.Existing octree leaves represent occupied voxels, thus the octree isused as memory efficient representation of a sparsely filled dense 3Dgrid.

In the example of FIG. 14, compression of a data representation forefficient transmission is achieved by the multi-level compression module416 through multiple levels, shown by the schematics 1410, 1412, 1414.The first level of compression (e.g., schematic 1410) is achieved byexploiting temporal and spatial redundancy within the accumulatingintegrated point cloud. Points are first quantized into configurablyrepresentative voxel occupancy; this has the effect of removingredundant points from consideration. The second level of compression(e.g., schematic 1412) is achieved during the encoding of the new voxelsproduced by change detection. The new voxels are added as leaves to anew octree (e.g., schematic 1414) whose spatial bounds match the forestoctree they inhabit. The adding of leaves to a new octree is considereda change. Since the change tree bounds match the forest tree bounds,both of the trees will match structurally as the exact branch nodes thatexist in the forest octree (e.g., 1414) to support the new leaves(voxels) will also exist in the change tree (e.g., 1412) to supportthose same leaves. Since the change is a true octree, it can be encodedinto a byte stream (e.g., schematic 1416) by an octree structureserialization method. In this method, a breadth-first traversal of allbranch nodes is performed and at each branch node visited, an 8-bitchild occupancy mask is generated where each bit corresponds to theoccupancy of the corresponding octant. The 8-bit child occupancy masks(bytes) are stored in order of generation, and the resulting byte arrayrepresents the complete structure of the octree. The disclosed 3Danalysis can also allow for transmission of video or hyper-spectraldata. For example, 3D classifications can help identify salient featuresin the data to allow more bits to be allocated to these regions duringcompression. FIG. 15 includes point cloud data representations 1510,1512, 1514, 1516, which illustrate an occupancy-based pruning example.Image 1510 shows point cloud reference data with 1.89M (M=million)points with 8 cm (cm=centimeter) resolution sampling given 75 k(k=thousand) points. Image 1512 illustrates a swatch of 316 k pointcloud that with temporal pruning has 12 k points. Image 1514 illustratesa swath of 309 k point cloud that with pruning has 9.5 k points. Image1516 illustrates each swath, colored by red (1518), green (1520), andblue (1522), respectively, showing overlap and additions within theoctree structure up to 8 cm. Another approach to data compression forstreaming point cloud data for visualization, which utilizes an XORrepresentation to octree changes, is described in J. Kammerl, Real-timeCompression of Point Cloud Streams, IEEE International Conference onRobotics and Automation (ICRA), 2012.

Visualization and Data Aggregation

Referring now in more detail to the live on-platform visualizationmodule 146 of FIG. 1, the illustrative live on-platform visualizationmodule 146 enables real-time visualization of the 3D map data, route andanalytics directly on a display device on the platform 120. Thereal-time visualization of the data collection on the platform 120allows a platform operator or other user to quickly verify the qualityand coverage of the sensor data 112, 114, 116 and then re-collect dataif necessary. The illustrative visualization module 146 thereby enablesusers to turn on and off layers of the real time data 112, 114, 116, andto manually annotate the map 132 with custom geo-tagged information ifneeded (in addition or as an alternative to the automated taggingprovided by the analytics subsystem 134 and the correlation module 136).The visualization module 146 also allows a user to leave geo-tags in theform of messages to other users. For instance, rather than simplyhighlighting a portion of the visualization with a tag, a user can ‘pin’a note to the area of interest on the visualization, alternatively or inaddition to other forms of geo-tags.

Referring now to FIG. 5, components of an embodiment of themulti-platform data aggregation subsystem 148 of FIG. 1 are shown inmore detail. Each of the components shown in FIG. 5 may be embodied assoftware, hardware, firmware, or a combination thereof. In FIG. 5, thecomponents of the multi-platform data aggregation subsystem 148 areshown in the context of an environment 501 that may be created duringthe operation of the computing system 100 (e.g., a physical and/orvirtual execution or “runtime” environment). The illustrativemulti-platform data aggregation subsystem 148 includes a livemulti-platform situational awareness module 510, a multi-platform datafusion module 516, a domain-specific data/maps data store 512, and anaggregated multi-platform data store 514. The live multi-platformsituational awareness module 510 allows users of the data aggregationsubsystem 148 to interact with the visualizations produced by thecomputing system 100 (in real time, in some embodiments). In someimplementations, the data aggregation subsystem 148 is embodied as acentral command center that can receive multimodal data streams frommultiple platforms 120 and fuse the multi-platform multi-modal datastreams into an aggregated map database 514 (e.g., by the multi-platformdata fusion module 516). The aggregated data 514 can be shared acrosseach of the platforms 120, e.g., as updates. Similarly, the aggregateddata 514 can be fused with additional, business-specific data 512 andbusiness logic can be applied to interpret and understand the integrateddata (by, e.g., the domain-specific data correlation module 136).Examples of such business logic include the planning of domain-specificrouting information or generating an appropriate response to a detectedfault, where the real-time information helps modify subsequent businessactions that are taken.

Increasingly, crowd-sourcing is being recognized as a valuable andpowerful tool for gathering data. Crowd-sourcing is supported in thereal-time system 100 through a mobile crowd-sourcing application 152that can directly send data points to the aggregated map database 514 inthe data aggregation subsystem 148. For example, a by a public network,a mobile user can mark a location on a map displayed on a display screenof his or her mobile device, and, with the crowd-sourcing app 152 selectfrom a list in the mobile crowd-sourcing app 152 issues or observationsthat he or she has made while viewing the data. Such a list can bedeveloped based on the business logic of information relevant to thespecific business process, e.g., a user can mark a location with apot-hole on a road utility app or take a picture of a damaged home aftera tornado for a disaster response app. The user selections may includethe GPS coordinates of the marked location and a picture of theobservation or other supporting information, in some embodiments.

The computational resources of the illustrative data aggregationsubsystem 148 enable large-scale 3D data visualization with layers ofreal-time data and higher-level metadata extracted either automaticallyor by a human user (e.g., a data analyst). The data aggregationsubsystem 148 is configured with an application interface that permitsextensive exploration of historical data, including change detectionbetween different collections and long-duration temporal analysis.

The data aggregation subsystem 148 can provide visualization of themulti-modal data 144 (which may be annotated data with geo-tags 138and/or markups 140) aggregated from multiple platforms 120, to users atlocations that are remote from the platform 120 (e.g., at a groundfacility or mobile device used to control the operation of the platform120). Alternatively or in addition, aggregated multi-modal data 144 maybe presented on the platform 120 via the live on-platform visualizationmodule 146. In addition to the automated analytics provided by the liveanalytics subsystem 134 and the correlation module 136, users of thedata aggregation subsystem 148 can manually nominateinformation/analytics to, for example, detect domain-specific anomaliesor changes, or to detect specific objects of interest. The resultinginformation can in turn can be provided to the analytics modules (e.g.,analytics subsystem 134, correlation module 136, used online or offline)for adapting algorithms for better performance. The user-specifiednominations can also be used as exemplars for changing the visualizationpresentation to accommodate user-specific preferences.

Embodiments of the platform 120 provide a point cloud streaming pipelinethat enables an interactive visualization system that is capable ofproviding a live and low-latency updated 3D representation of theimmediate area surrounding the platform 120, as well as areas theplatform 120 has already visited, in one geometrically consistentmodel/map. In some cases, the platform 120 provides remote platformoperators with an improved ability to navigate and explore environments(including hostile environments), which is superior to that which can beachieved by the limited camera systems currently in use.

Some embodiments of the data aggregation subsystem 148 utilize, e.g.,graphics processing unit (GPU) hardware-accelerated OpenGL for renderingthe visualization. The medium of presentation for theenvironment-representative 3D model is a point cloud visualization,where each point is derived from the centroids of all voxels in themodel (and possibly associated metadata). The specific technique foroptimized OpenGL streaming buffer updates can be selected based onrequirements of a particular implementation of the platform 120. In someimplementations, it may be desirable to limit OpenGL central processingunit (CPU)-to-GPU implicit synchronizations for better performance.

In some embodiments, the visualization output includes a number ofdifferent colors, since drawing individual points as single white pixelswould be ineffective for visually differentiating differentenvironmental structures from one another. The visualization technologyutilized by the visualization module 146 and/or the data aggregationsubsystem 148 outputs data according to a color scheme and/or providesthe user with a toolbox of point coloring schemes and interactivelyadjustable parameters (such as max colored range from virtual camera),which may be selected (e.g., toggled) as needed for various specificspatial awareness tasks. The different available color-coding modesinclude, for example: color gradient by height, color gradient bydistance to selected point, color linear by distance to selected point,color by distance to virtual camera, color linear by distance to camera,color by normal orientation, color by detected planes, and color byprojected camera imagery. Whereas changing the actual color propertiesof points by adjusting a GPU-resident vertex data would be very slow,the disclosed visualization technology utilizes vertex and fragmentshaders (written in GLSL) to allow toggling between color modes on thefly, without ever touching the GPU-resident data. In some cases, allcolor generation is performed via a combination of vertex and shadersthat run on the GPU. This method allows rapid toggling betweencolorization modes, by just switching the active shaders. This method isextremely fast, and is further optimized by utilizing the GLSLsubroutine system.

Navigation within the point cloud is also provided by the disclosedvisualization technology. In some embodiments, to facilitate relativesensor platform to point cloud spatial understanding, a virtual proxymodel of the platform (such as a Talon robot) is rendered within thepoint cloud. The position and orientation of the virtual proxy modelmatch those of a real world platform with respect to the point cloud.This method also allows the operator to switch between 3 base navigationmodes: an orbit camera mode, which allows the operator to select anypoint within the entire point cloud and interactively yaw, pitch, andzoom the virtual camera about that point; an FPS flight mode, whichallows the operator to fly the virtual camera as if it were an aircraft(that can hover), interactively adjusting speed; and a follow cameramode, which allows the operator to yaw, pitch, and zoom the virtualcamera about the platform proxy, utilizing a simulated elastic tether(based on a critically damped spring model) between the platform proxyand the virtual camera, as the platform moves throughout the point cloudwhere the virtual camera is smoothly pulled along with it. Theimportance of point colorization and navigation is to aid the operatorwith comprehending the spatial distribution of the rendered points.Since the points are drawn on a 2D surface, depth cues are provided toaid in the operator's understanding of the points' relative placementwithin 3D space. This is accomplished through colorization by alteringpoint color based on spatial parameters such as: distance from ground,distance from camera, distance from platform, planarity of a surface(via normal colorization), etc. This is also accomplished by parallax,via intuitive movement of the virtual camera in respect to the pointcloud. Examples of colorization modes and visualization features areshown in FIG. 17 of the aforementioned U.S. Provisional Application Ser.No. 61/954,635, filed Mar. 18, 2014, entitled “Real-Time System forMulti-Modal 3D Geo-Spatial Mapping, Object Recognition, Scene AnnotationAnd Analytics.”

6DOF Pose Estimation

Referring now to FIG. 6, components of an embodiment of the 6DOF poseestimation module 218 of FIG. 2 are shown in more detail. Each of thecomponents shown in FIG. 6 may be embodied as software, hardware,firmware, or a combination thereof. In FIG. 6, the components of the6DOF pose estimation module 218 are shown in the context of anenvironment 601 that may be created during the operation of thecomputing system 100 (e.g., a physical and/or virtual execution or“runtime” environment). In the illustrative 6DOF pose estimation module218, an “IMU-centric” framework for doing accurate 6DOF pose estimationeven with lower-grade IMUs (such as MEMS IMUs) includes amotion/location/orientation sensor (e.g., “IMU”) mechanizationprocessing module 610, a visual feature detection and matching module612, a relative pose estimation module 614, a feature track measurementmodule 616, and an error-state extended Kalman filter 618. Thecomponents of the 6DOF pose estimation module 218 utilize bothmotion/orientation/location data and visual features extracted from 2Dand/or 3D imagery to generate 6DOF pose estimates.Motion/location/orientation data stream(s) 116 are input to the IMUmechanization processing module 610. The IMU mechanization processingmodule 610 generates a motion model 622. The visual features detectionand matching module 612 receives 2D and/or 3D image sensor data streams112, 114 and generates feature and match data 624. The relative poseestimation module 614 utilizes the motion model 622 and the feature andmatch data 624 to generate feature tracks 626. The feature tracks 626are used by the feature track measurement module 616 to generate local(relative) track measurements 628, which are used by the error-stateextended Kalman filter 618 to generate 6DOF pose estimates 630. The IMUmechanization processing module 610 utilizes the 6DOF pose estimates 630to generate corrected navigation data 640 (e.g., an estimated navigationpath for the platform 120), which can be used by other modules of theplatform 120 as described herein.

In more detail, the illustrative Kalman filter module 618 utilizes anerror-state (e.g., indirect) form of the Kalman filter in order to avoidthe need for dynamic modeling of the complex kinematics associated witheach specific sensor 102, 104, 106. In the pose estimation module 218,the motion model 622 is derived from integrating the gyroscope andaccelerometer data from the IMU (as performed in the IMU mechanizationprocessing module 610) to form the building block for state prediction,rather than explicitly modeling the platform dynamics and using the IMUon the measurement side. Since the IMU is able to follow very accuratelythe high frequency motion of the platform, the indirect Kalman filteroperates on the inertial system error propagation equations, whichevolve smoothly and are much more adequately represented as linear. Anadvantage of this approach is that the navigation solution can betransferred to another mobile platform (carrying equivalent sensors)without any changes to the sensor fusion algorithms.

As each new frame is received from data stream(s) 112, 114, visualfeatures are extracted by the feature detection and matching module 612by use of, e.g., a Harris corner feature detector. These features arematched by the feature detection and matching module 612 to thepreviously-received frame using, e.g., normalized correlation of imagepatches around each such feature. Match success or failure is determinedbased on, e.g., mutual agreement criterion. In some embodiments, the6DOF pose estimation module 218 does not use any thresholding ofcorrelation scores, rather it decides that a match is successful if bothfeatures in the current and previous frame pick each other as their best(highest scoring) matches. A feature track table that accommodatesseveral hundred frames is used to record the feature locations andmaintain the track information and track lengths. After the initialmatching step, there are usually a great number of outliers due to falsematches. In the pose estimation module 614, all features that aretracked for at least three frames (e.g., feature and match data 624) areinput to a robust two-point relative pose estimation algorithm thatenforces geometric constraints across three frames. Following thisprocess, inlier feature tracks (e.g., feature tracks 626) are determinedbased on, e.g., the comparison of trifocal Sampson error for eachfeature against a predetermined threshold. Those tracks that fail thistest are terminated and reset as fresh features that were newly detectedat the current frame, as these features may become inliers in thefuture.

At every video frame of the data stream(s) 112, 114, the entire featuretrack history with inlier tracks from the current frame extending to thepast frames in the sequence is made available to the Kalman filter inthe error-state extended Kalman filter module 618. A separatemeasurement equation is created for each feature that is tracked formore than three frames. After this step, the measurement equations forall the tracks are stacked to form the final set of measurement modelequations which are a function of both the previous state and thecurrent predicted state, so they are relative (local) in nature asopposed to more typical global measurements which are a lot morestraightforward to treat. In order to properly handle such relativemeasurements in the extended Kalman filter, in some embodiments, astochastic cloning framework is employed. One approach to handlingrelative measurements between the previous and current time instants isto express them in terms of motion estimates between the two states andtake into account the cross correlation arising from such a formulation.Stochastic cloning provides the framework to process such relativemeasurements by augmenting the state vector with two copies of the stateestimate, one evolving and one stationary clone. The evolving clone ispropagated by the process model (similar to a conventional Kalman filterframework), whereas the stationary clone is kept static and does notevolve. The relative measurement between the previous and current timeinstant is then expressed as a function of these two states, and aKalman filter update, modified to incorporate the joint covariance ofthe two clone states, is performed.

Similar to the video based features, LIDAR based features can beintegrated and tracked to provide measurements to the Kalman filter inthe error-state extended Kalman filter module 618. However, in the caseof scanning LIDAR systems, the initial Kalman pose estimates (e.g., 6DOFpose estimates 630) are used to locally integrate the LIDAR data basedpoint to enable feature extractions. Salient features such a 3Dspin-images can be utilized for tracking these features.

In some embodiments, an addition to the local Kalman filtering process,simultaneous localization and mapping (SLAM methods) can be used toimprove precision while developing a navigation map. Embodiments of thedisclosed system 100 utilize visual features such as HOG and SIFTdescriptors for establishing longer range correspondences. Similarly,with LIDAR data, ICP (iterative closest point) methods can be used toestablish longer range matches across swaths of locally integrated data.The longer range methods can also exploit the GPS data, DEM (digitalelevation model) data and other sensor sources in the joint inferenceprocess.

Object Recognition and Classification

Referring now to FIGS. 7A, 7B, and 8-12, components of an embodiment ofthe object/scene recognition and classification subsystem 312 of FIG. 3are shown in more detail. FIG. 7A shows a functional architecture of ascalable object recognition system for detecting and identifying a largeclass of objects, even in cases where some objects can only beidentified using the context of the full scene. Each of the componentsshown in FIG. 7A may be embodied as software, hardware, firmware, or acombination thereof. In FIG. 7A, the components of the object/scenerecognition and classification subsystem 312 are shown in the context ofan environment 701 that may be created during the operation of thecomputing system 100 (e.g., a physical and/or virtual execution or“runtime” environment). As described below, the illustrativeobject/scene recognition and classification subsystem 312 includesautomated tools for feature and attribute extraction, object detectionand scene classification. The recognition and classification subsystem312 includes both “online” and “offline” functionality. Those modulesthat are typically implemented for online operational use (e.g., in aninteractive, real-time operational environment of the platform 120) aredenoted by the dashed box 702. The modules that are typicallyimplemented for offline use (e.g., in the background or as part of theprocess of developing/maintaining the functionality of the online system702) are denoted by the dashed box 703.

The illustrative online functionality 702 includes a large scale objectand region extraction module 710, a feature computation module 718, acontext-free object identification module 726, and a contextual objectidentification module 732. The illustrative offline functionality 703includes a visual words computation module 738, an object classifiertraining module 740, and a contextual relationship determination module742. One or more of the components of the offline functionality 703 maycommunicate with one or more of the components of the onlinefunctionality 702 from time to time, e.g., to provide model updates tothe components of the online functionality 702.

The illustrated architecture provides a layered object recognitionframework for the recognition of a set of object and scene classes(where a “class” may refer to a descriptive label, such as a type orcategory of objects). The layered framework progresses from recognitionof simple to complex classes, from large scale structure recognition tofiner scale recognition, and from independent object level recognitionto recognition with contextual inferencing. Object extraction andidentification are intertwined in an iterative computation in whichprogressive disambiguation of class labels is used to refine extraction,which in turn improves the accuracy of the object identification.

The illustrative architecture of the recognition and classificationsubsystem 312 realizes the object recognition algorithms as a scaleablesystem for large area classification with an extensible set of classes.In some embodiments, efficient data structures and algorithms for largescale handling of spatial data, image data, multi-dimensional featuresand learning classifiers are implemented using off-the-shelf systems. Asa result, the disclosed architecture supports easy interfacing withusers of the technology, and enables the recognition and classificationsubsystem 312 to transition to a real-time 3D reasoning system ifdesired.

In some applications, urban infra-structural objects, from dumpsters toalleys, are important for tactical operational planning. The disclosedtechnology (e.g., offline technology 703) can be configured to identifya comprehensive collection of urban objects. In some embodiments, objectrecognition technology is used to automatically extract and identifynumerous urban and semi-urban classes of tactical relevance with 3DLIDAR and 2D EO (electro-optical) sensor data of city scaleenvironments. This 3D urban object recognition technology can bedeployed in, among other applications, real-time 3D query and reasoningsystems.

The illustrative recognition and classification subsystem 312 isembodied as a layered end-to-end recognition system that understands ascene by extracting and identifying progressively complex objectclasses. Easy-to-detect, distinctive large-scale objects (e.g., ground,foliage, roofs, walls, and buildings) are recognized first as a coarselayer. Recognition of these large-scale objects exposes compact regionsof space that are supported by the recognized objects, which are, e.g.,the ground, a roof or a wall. The regions of space defined by therecognized larger-scale objects are extracted as “proto-objects” and maytypically contain one or more as-yet unidentified classes of objects.Proto-object extraction provides a focus-of-attention mechanism forefficiently handling complexity in a large cluttered scene. Objects inthese proto-object regions are progressively identified, applyingmultiple hypotheses to ambiguous objects. Some context-free objects(e.g., a crane, or a satellite dish) with discriminative, invariantfeatures are identified first without contextual reasoning (where“context-free” may indicate that the object can be identified accuratelywithout reference to the particular context in which it occurs).Context-sensitive objects (e.g., doors, parking area) are identifiednext utilizing 3D spatial relationships (where “context-sensitive” mayindicate that an object's identification accuracy can be improved by theuse of context information, such as surrounding or neighboring objectsor landscape). Finally, compound objects (e.g., gas stations, whichinclude buildings, gas pumps, parking areas, etc.) are identifiedthrough, e.g., bag-of-object characteristics (loose configuration ofsimpler objects co-located).

Using the layered architecture of the recognition and classificationsubsystem 312, early recognition of large scale and easily recognizableobjects occurs pre-attentively and quickly, and sets the stage forrecognizing more complex objects within a context. Using 3D and 2D datain a combined unary and contextual recognition approach avoids“inattentive blindness,” where human visual systems miss conspicuousevents. In contrast with one-filter-per-class and one-shotclassification approaches, some embodiments of the disclosed approachcan avoid errors by entertaining unique or multiple hypotheses as thedata may suggest.

In the illustrative architecture, object recognition progresses in acontinuous manner from large-scale structures to fine scale recognitionand from independent context-free recognition to recognition withcontextual inferencing. This architecture utilizes a multi-hypotheses,non-committal strategy in which early, coarse scale ambiguousrecognition is successively refined with finer scale data and contextthat leads to progressively more unambiguous object recognition.

The illustrative large scale object and region extraction module 710utilizes a 2D extraction module 712 and a 3D extraction module 714 toperform large-scale object extraction and initial proto-objectextraction. In some embodiments, the large scale object and regionextraction module 710 handles LIDAR scans by sub-dividing the scans tocover in the range of about 1-5 square kilometers of urban locales. Theextraction module 710 handles the challenge of rapid area delimitationby first extracting and identifying “easy” objects recognizable aslarge-scale coarse entities, such as ground, foliage, roofs, walls, andbuildings. This is accomplished through use of low-level 3D features:local planes, 3D scatter matrices, and tensor voting, and efficient datastructures for the representation of occupied and free 3D space.Extraction and identification of the ground plane, buildings and similarlarge-scale structures achieves at least two goals. First, it providescontext for finer scale object recognition, and second, it enablesintelligent search to efficiently handle non-uniform space occupancy.

The extraction module 710 uses contextual frames of reference, such asground and buildings, to extract isolated regions of space, referred toherein as proto-objects, which may contain multiple objects that are asyet unidentified. Finer scale objects within the proto-object regionsare extracted and identified in subsequent modules of the objectrecognition and classification subsystem 312. The extractedproto-objects provide focus-of-attention for efficiently handling thecomplexity of a large-scale 3D scene.

The goal of the large-scale and initial proto-object extractionperformed by the extraction module 710 is to rapidly analyze large areasof the scene using low-level cues such as planarity and perceptualgrouping to generate 3D regions that contain objects of interest withhigh probability. To do this, the components of the extraction module710 execute a series of algorithms, such as those described below.

1. Ground Extraction. The ground (e.g., earth, floor, roadway, etc.) isextracted using local vertical and planarity constraints with 3D datasampled at a coarse resolution. 3D points with z-component (local updirection) close to the local minimum ground height are retained asputative ground points. Local surface normals are computed at voxelswith these ground points. Ground regions are grown from these voxels byrecursively connecting neighboring voxels with consistent local planeestimates. This ensures a sharp delineation of the 3D objects that donot belong to the ground. A parametric model is finally fit to obtain aterrain model. Ground points are removed for the subsequent steps.

2. Off-Ground Local Surface Classification. At each voxel, theillustrative extraction module 710 adaptively performs an eigenvaluedecomposition of the local 3D scatter matrix at multiple scales, thusensuring a stable fit, while preserving local 3D discontinuities. Largervoxels are used to extract big structures such as buildings, whilesmaller voxels are used for other objects. Voxels are labeled with therank of the scatter matrix as: locally planar (rank 1), curvilinear(rank 2), or 3D texture (rank 3).

3. Proto-Object Extraction. Rank 1 and rank 2 tensors are grouped overlarger 3D regions using perceptual organization. Coherent 3D surface(rank 2), or wire-like regions (rank 1) are grouped into larger regionsbased on proximity and consistent surface normals or tangent directions,respectively. In some implementations, proto-objects are 3D regions thathave a high likelihood of belonging to a part, or a whole object, thatis to be identified in subsequent layers. The relationship between theseregions is encapsulated in an adjacency graph, which will be used duringcontextual object identification performed by the contextual objectidentification module 732, described below. The Off-Ground Local SurfaceClassification and the Proto-Object Extraction are repeated atprogressively finer resolutions as needed to extract smallerproto-objects.

4. Building Extraction and Vegetation Clutter Removal. Buildings areextracted within 3D regions using region descriptors such as: (i)histogram of angles between the normals of a region and the local groundnormal; (ii) histogram of angles between normals of the region and theeasting direction; (iii) histogram of projections of 3D points in aregion along the ground direction; (iv) first and second order momentsof 3D points within a region; (v) area of the region. These descriptorsare used to train a classifier to label regions as: “clutter,” “wall”(large, vertical surface), “roof” (large, horizontal or tilted surface),and “proto-object.” Priors, such as heights of roofs, size of buildingwalls are also incorporated into the classifier. Neighboring regionscontaining a roof and walls are further labeled as building candidates.The 3D orientation of each building and extracted proto-object areestimated from the region descriptors and projected into thecorresponding 2D images to extract 2D object regions.

The coarse-to-fine processing described above enables an efficientcomputation of distinctive coherent surfaces and rapid rejection ofvegetation clutter, which lacks the smoothness of artificial objects. Italso enables the inference of relationships between the proto-objects.In FIG. 12, an example 1200 shows results of building and groundextraction and foliage removal that have been achieved with 50 aerialdata. In the example 1200, the image 1210 depicts the original pointcloud; the image 1212 depicts a 2D planar surface grouping; the image1214 depicts identification of planar structures (1216) vs. foliage(1218); and the image 1222 depicts identification of buildings (1224)vs. ground (1226). Other examples of large-scale and proto-objectidentification are provided in FIGS. 7 and 9 of the aforementioned U.S.Provisional Application Ser. No. 61/954,635.

Referring again to FIG. 7A, the feature computation module 718 performsfeature computation for object recognition using the extracted objectsand regions 716 output by the extraction module 710. Proto-objectsextracted by the extraction module 710, the first layer, are recognizedusing modules the feature computation module 718, the context-freeobject identification module 726, the contextual object identificationmodule 732, and the complex object identification module 736 insubsequent layers. The feature computation module 718 utilizes extractedobjects and regions 716 produced by the extraction module 710 togenerate a representation of objects in terms of data-driven parts. Todo this, the feature computation module 718 uses invariant semi-local 3Dand 2D features produced by, e.g., 3D feature computation module 722 and2D feature computation module 720. The invariant 2D and 3D featuresenable the following: (1) objects can be recognized under occlusions andmissing data, without the need for full geometric relationships betweenparts; (2) invariant features handle pose variations and viewingdistances; (3) multi-scale features work with a wide range ofresolutions present in aerial and ground collections; (4) clusteredfeature prototypes, called 3D shapemes, 2D visemes and 3D/2D textons,succinctly capture intra-class and inter-class variations for efficientobject recognition.

Some examples of features for object recognition include invariant 3Dshape histograms, such as spin images and shape contexts, 2D shape andappearance features such as histograms-of-gradients, steerable pyramidsand Gabors, and texture features such as textons. In some cases, jointexploitation of 3D and 2D data is performed to fix the scale andperspective of 2D images for view invariant 2D feature computation.Occlusion reasoning in 3D can be used to infer missing object parts.

The feature computation module 718 utilizes semi-local, scale and poseinvariant 3D and 2D features for object recognition. The 3D/2D shape andappearance (shapemes, visemes) and 3D/2D texture (textons) featuresprovide robustness of recognition as well as efficiency of computation.These features are resilient to missing data, pose and view variations,partial occlusions and the presence of unknown classes as clutter andcan handle intra- and inter-class appearance and shape variability.Generalizability with features enables recognition so that precisemodels for each class are not required. The semi-local discriminativefeatures used by the feature computation module 718 for data-drivenpart- and sub-part based description of objects are designed to capture3D and 2D shape, appearance and texture properties of a wide range ofobject classes.

Embodiments of the feature computation module 718 can: (1) work directlywith point cloud data without the need for polygonal models of objectsof interest, thereby enabling easy extension of the set of targetedfeatures without expensive modeling and learning; (2) apply invariantsurface and feature representations such as spin images, or the verticalsupport histogram, which are resilient to surrounding clutter, missingand incomplete data due to occlusions, and non-uniform densities inpoint cloud capture; (3) achieve precision alignment of 2D and 3D shapesfor verification of features for ensuring high accuracy and low falsealarms. Semi-local, multi-scale, invariant 3D and 2D features helpaddress data and model issues, such as: robustness to partialocclusions, pose invariance, handling variable resolution and precision,resilience to clutter, part-based recognition, and intra- andinter-class variability. Additionally, the invariant 3D and 2D featurescan represent significant and salient components of an object withoutthe need for precise segmentation into semantic parts. Instead, thepresence of features in flexible geometric configurations is sufficientfor robust recognition.

FIG. 8 depicts an example of a coarse-to-fine scene processing andclassification system 801, which can be used for large scale objectrecognition by the extraction module 710. At 812, ground extractionalgorithms are performed on LIDAR data 810 to classify the LIDAR data810 as either ground 814 or non-ground 816. Terrain modeling 818 isperformed on the ground data 814, and the output of terrain modeling 818is processed by clutter removal algorithms 820 and/or building modelingalgorithms 830. The nonground data 816 is processed by one or more of:the clutter removal algorithms 820, roof classification algorithms 822,wall classification algorithms 824, boxy object detection algorithms826, and wiry object detection algorithms 828. Output of any of thesealgorithms can be further processed by other algorithms, as denoted inFIG. 8 by the bidirectional arrows. As a result of roof classification822 and building modeling 830, one or more buildings 838 may beidentified in the ground or nonground LIDAR data, or the data may befurther processed, e.g., by the clutter removal algorithm 820 or thewall classification algorithm 824. Application of wall classificationalgorithm 824 may result in, for example, detection of a fence 832, orfurther processing of the LIDAR data by the roof classification 822 orthe boxy object detection 826. Application of boxy object detection 826may result in detection of, e.g., cars or a mailbox 834, or furtherprocessing of the LIDAR data by wall classification 824 or wiry objectdetection 828. Application of wiry object detection may detect poles,wires, etc. 836, or may result in further processing by boxy objectdetection 826.

Referring again to FIG. 7A, the illustrative context-free objectidentification module 726 performs context-free object identificationwith feature sharing, using the 2D and 3D features 724 extracted by thefeature computation module 718. The context-free object identificationmodule 726 uses features for joint learning of multiple classes, usingfeature sharing and context in the proto-object identification layer.This enables the system 100 to used limited exemplars and models formulti-class object recognition and for recognition in context.Classification with feature sharing can be performed using, e.g., anindexable database of joint features for a large database of objectssuch as vehicles. The context-free object identification module 726utilizes multiple techniques to handle a large class of objects,including, for example: (1) automated learning of shared featuresamongst classes using a logarithmic number of features as a function ofthe number of classes; and (2) a visual vocabulary of feature prototypesto create clusters of similar classes for linear-in-number-of-classescomplexity for learning.

FIG. 9 illustrates an example of how feature sharing amongst objectclasses may proceed (in the direction from left to right on thedrawing). Images of a lamp post and a telephone pole have many similarfeatures, while a door is distinctly different and does not share manyfeatures with the lamp post or the telephone pole. For instance, thefeature 910, compact object, may be shared by all of the lamp post 932,telephone pole 934 and door 936 images. The features 912, 916, 922 mayall be applicable to a pole 928, which could be either a telephone poleor a lamp post. However, the feature 920, oval top, is applicable to alamp 926 but not to other types of poles. The features 914, 918, 924 areapplicable to doors 930 but not poles 928 or lamps 926.

Referring again to FIG. 7A, the illustrative contextual objectidentification module 732 performs principled contextual inferencing.The principled approach to object classification is used withinter-class and spatial constraints expressed as graph models withbelief propagation. This approach enables learning context both fromexamples as well as priors derived from generic world knowledge andhuman specified rules. FIG. 10 depicts an example 1000 of the use ofcontext information and belief propagation to classify smaller-scaleobjects. Proto-object classification 1018 of objects depicted in images1010, 1012, 1016 may identify an upright object as either a fire hydrant1024 or a post box (mailbox) 1026. The proto-object classification isused to initialize a graphical model 1020, with which belief propagation1022 is performed, resulting in a final object identification of firehydrant 1028. The classification of “fire hydrant” may be based on, forexample, 3D neighboring objects that have already been classified. Forexample, if the system 100 has already identified a building and astreet, these object identifications can be used to influence theclassification of the unknown object. FIG. 11 further illustrates theuse of context information and belief propagation to classify objectsbased on the relationships of the objects to one other. In FIG. 11, therelative distances between the object classes (e.g., C1, C2, C3, C4) areused to compute a potential between each of the connected nodes.Connected nodes exchange messages m, which are combined with localevidence to compute a belief at each node. The computed belief leads toobject classification.

In order to recognize some object or scene classes, such as an alley, ora factory, the recognition and classification subsystem 312 uses contextfor object extraction and identification. The interaction between objectidentification and extraction and bag-of-objects identification can beused to improve accuracy in object recognition. Graphical models can beused for the representation of context, and belief propagation can beused as a principled method for inferencing with context. Contextualinferencing can be extended to include classes that can be looselyrepresented as a bag-of-objects. These classes can represent complexobjects, such as a mobile home park, a factory, or a railroad yard. Thecomplex objects may be a conglomeration of object types arranged inflexible geometric relationships.

The illustrative recognition and classification subsystem 312 performscontext-free object identification and contextual object identificationiteratively. The context-free object identification module 726 generatesobject class hypotheses 728, which are used by the contextual objectidentification module 732 to produce context-based extracted objects730, which are then fed back to the context-free object identificationmodule 726.

Referring still to FIG. 7A, the illustrative complex objectidentification module 736 performs bag-of-objects identificationtechniques to identify complex objects. The context-free and contextualidentification layers (performed by, e.g., the context-free objectidentification module 726 and the contextual object identificationmodule 732) identify and extract objects typically in connected, compactregions of space. The next processing layer, performed by the complexobject identification module 736, addresses the recognition of complex,extended object classes such as a golf course, processing plant, ormobile home park. These complex objects consist of a loose geometricconfiguration of other small objects, and are called bag-of-objects. Forexample, individual units of a mobile home park can be at variousdistances from each other. Embodiments of the complex objectidentification module 736 can employ the following types of features toidentify and extract these complex objects: (i) textons, forcharacterizing texture and appearance in 3D and 2D date acquired mainlyfrom the air. For example using textons large geographical areas can becharacterized as being a grassy surface, forest, crop field, or an urbanenvironments; (ii) “bag-of-objects” descriptors. Bag-of-objects is akinto the popular bag-of-features descriptor used in computer vision andinformation retrieval for retrieving images, video and documents usinghistograms of visual words and textual words. Geometrical relationshipsbetween visual words are ignored with the idea that visual words arerich enough to capture class specific characteristics. Thebag-of-objects descriptor is a histogram of object classes, which occurwithin an area. The descriptor does not capture the relationship betweenthe objects, but can be used for recognizing semantic objects that havelittle or no constraints between the components such as a factory orfeeding lot. In essence, the bag-of-objects descriptor captures the gistof regions within a scene, i.e., it specifies the individual content,but not the relationship between components; (iii) contextualbag-of-objects descriptor, which consists of concatenated bag-of-objectshistograms of recognized small objects within regions centered at a 3Dlocation. The contextual bag of objects provides more discriminationcompared to the simpler, bag-of-objects feature, yet is flexible toconfiguration changes between the individual objects. For example,objects such as a gas station have a constrained spatial distributionbetween the objects (pumps occur at a certain distance from thebuilding). The foregoing features, designed to capture thecharacteristics of complex objects, are employed in a classificationframework that utilizes training data from annotated scenes as well asfrom pre-defined rules is used. The rules can be in the form of classespresent as well as sketches of rough arrangements of classes.

Training and context learning are typically performed offline by thevisual words computation module 738, the object classifier trainingmodule 740, and the contextual relationship determination module 742.The components of the offline functionality 703 develop multi-classclassifiers learning classifiers with limited exemplars, e.g.,classifiers that automatically and optimally share features acrossclasses and can work with limited training examples. This approach canreduce error in classification by not committing to a labeling on aclass-by-class basis but rather keeping hypotheses alive until morefeatures from later layers lead to more specificity. The classifiers areconstructed automatically and require limited training data. Theclassifiers naturally achieve generalization and robustness tointra-class variations. New classes can be incrementally added.

Referring now to FIG. 7B, an exemplary plot 704 illustrates theheterogeneity of object classes and the disclosed layered approach forexploiting that heterogeneity. The classes range (along the “y” axis)from smaller-scale objects, such as fire hydrants and trash-cans, tolarger-scale objects, such as cranes and buildings, to complex objects,such as streets, golf courses and trailer parks. The objects are rankedalong the context (“x”) axis. Categories such as buildings, trees, andground may be capable of being recognized with very little context whilecomplex (e.g., compound) objects such as factories and golf courses mayonly be recognized by detecting the presence and co-location of a largenumber of smaller object classes (using, e.g., a bag-of-objects).

As noted above, principled contextual inferencing can be used to exploitscene context using the identity of objects within local spatialregions. For example, independently distinguishing small objects such asfire hydrants from other small objects, such as post boxes, may bedifficult. However, context information relating them to larger, moresalient objects may disambiguate the exact class identity of smallobjects, e.g., fire hydrants occur close to buildings. Contextualinformation is encoded as a graphical model where each node encapsulatesthe belief that the object belongs to a class. Belief propagation isused to refine and detect object classes.

Referring now to FIG. 16, an example of a process 1600 implemented ascomputer-readable instructions executed by the computing system 100 toperform multi-sensor, multi-modal 3D geospatial mapping, objectrecognition, scene annotation and analytics is shown. The process 1600may be embodied as computerized programs, routines, logic and/orinstructions executed by the platform 120 of the computing system 100,for example by one or more of the modules and other components of theplatform 120 shown in FIGS. 1-6 and 7A, described above. At block 1610,the system 100 receives sensor data from the multiple, multi-modalsensors (e.g., sensor data 112, 114, 116 from sensors 102, 104, 106). Inblock 1612, the system 100 temporally aligns or synchronizes the variousmulti-modal data streams received in block 1620. In block 1614, thesystem spatially aligns the synchronized sensor data, using, e.g.,spatial and geo-spatial association techniques. In block 1616, thesystem 100 generates a navigation path and geospatial map for theplatform 120. To do this, the system 100 may perform multi-sensor 6DOFpose estimation in block 1618, using, e.g., an error-state extendedKalman filter as described above with reference to FIG. 6. In block1620, the system 100 performs geospatial data integration of themulti-sensor multi-modal data (e.g., sensor data 112, 114, 116).

The data integration performed in block 1620 may result in an integratedmap representation of the sensor data, which can be visualized oranalyzed by, e.g., plug-in style analytics and/or business logic. Inblock 1622, the system 100 determines whether a live analytics module(e.g., a live analytics subsystem 134) is available. If no liveanalytics are available, the system proceeds to block 1630 to performcompression of the integrated sensor data for visualization and/ortransmission to another device. If live analytics are available, thesystem 100 proceeds to block 1624. In block 1624, the system 100performs live analytics on the integrated multi-modal sensor data andannotates the integrated multi-modal sensor data. To do this, the systemperforms object/scene recognition and classification, change detection,and/or anomaly detection algorithms to identify useful annotations, andadds the annotations to the map representation of the integratedmulti-modal sensor data.

In block 1626, the system 100 determines whether any domain-specificbusiness logic (e.g., a domain-specific data correlation module 136) isavailable. If no business logic is available, the system proceeds toblock 1630 to perform compression of the integrated sensor data forvisualization and/or transmission to another device. If business logicis available, the system 100 proceeds to block 1628. In block 1628, thesystem 100 executes the domain-specific business logic on the integratedmulti-modal sensor data and annotates the integrated multi-modal sensordata with markups produced as a result of the application of thedomain-specific business logic. To do this, the system may performdomain-specific object/scene recognition and classification, changedetection, and/or anomaly detection algorithms to identify usefuldomain-specific markups, and adds the domain-specific markups to the maprepresentation of the integrated multi-modal sensor data.

In block 1630, the system 100 creates a compressed version of theannotated and marked-up (as needed) map representation of the integratedmulti-modal sensor data. To do this, the system 100 may employ one ormore of the approaches described above with reference to FIGS. 4 and14-15. Following the data compression performed in block 1630, thesystem 100 may proceed to block 1632 and/or block 1634. In block 1632,the system 100 presents a live visualization of the annotated (andmarked up, as needed) integrated multi-modal sensor data locally, e.g.on the platform 120. In block 1634, the system 100 aggregates thelocally produced annotated sensor data with similar data produced byother platforms 120, and presents a visualization of the aggregated dataon a display device. This data aggregation and visualization of theaggregated data may be performed on the local platform or on anothercomputing device, such as a ground-based command center or mobiledevice.

Referring now to FIG. 17, an example of a process 1700 implemented ascomputer-readable instructions executed by the computing system 100 toperform object recognition and classification is shown. The process 1700may be embodied as computerized programs, routines, logic and/orinstructions executed by the computing system 100, for example by one ormore of the modules and other components of the computing system 100shown in FIGS. 1-6 and 7A, described above. The illustrative process1700 includes steps or subprocesses contained in box 1702 that canexecute concurrently, e.g., by multiple processors, as described belowwith reference to FIG. 18. The process 1700 also includes steps orsubprocesses contained in box 1712, which can execute iteratively.

At block 1710, the system 100 performs large-scale and/or proto-objectrecognition on the visual imagery captured by, e.g., 2D and/or 3D imagesensors coupled to or integrated with the computing system 100. To dothis, the system 100 may utilize any of the techniques described abovewith reference to the extraction module 710 of FIG. 7A. In block 1714,the system 100 performs context-free object identification using thelarge-scale object and/or proto-object information generated in block1710. To do this, the system 100 may utilize any of the techniquesdescribed above with reference to the context-free object identificationmodule 726 of FIG. 7A. In block 1716, the system 100 performs contextualobject identification (e.g., contextual inferencing) utilizing theobject identification information generated in block 1714. The system100 may iterate between blocks 1714, 1716 to further classifysmaller-scale objects detected within the large-scale or proto-objectareas detected in block 1710. In block 1718, the system 100 utilizes thesmaller-scale object identification information produced by block 1716and/or block 1714 to perform complex object recognition. To do this, thesystem 100 may utilize a bag-of-objects algorithm as described abovewith reference to the complex object identification module 736 of FIG.7A. The resulting output of the process 1700 can be used in thegeneration of a map representation for, e.g., the platform 120 or forfurther analysis (including change detection and/or anomaly detection asdescribed above).

EXAMPLE USAGE SCENARIOS

Many various applications of the multi-sensor multi-modal datacollection, analysis, recognition, and visualization technologiesdisclosed herein are possible, including applications addressed toautomated collection and analysis of geospatial data for facilitymonitoring, asset protection, and monitoring or surveillance of naturalresources (e.g., water supplies, agricultural properties, forests, etc.)or man-made structures, including public utility networks (e.g.,electric, gas, oil, water, cable), dams, levees, roads, bridges,tunnels, etc.

In one implementation, the data representation, compression, andvisualization modules were assessed qualitatively and quantitativelyusing data collected by a multi-sensor rig mounted on a Segway RMP(model=200). Evaluation was performed using two different sensorconfigurations that reflect common operational configurations for groundrobots. Both configurations used LIDAR sensors paired with stereocameras and a low-cost MEMS IMU. The cameras and IMU form a navigationsolution that can provide reliable 6-DOF pose in GPS-enabled orGPS-denied environments. The first sensor configuration paired a singlelaser scanning LIDAR (the Hokuyo UTM-LTX), mounted coronally on a rotarystage that repeats a constant velocity 180 degree pan behavior, thuscollecting a 360 degree point cloud with full hemispheric field-of-view.The Hokuyo sensor receives range measurements, in the form of line scanswith 1081 samples (0.25 degree angular resolution), at 40 Hz. The secondsensor configuration used the Velodyne HDL-32e, a multi-laser LIDAR thathas been extensively used for autonomous vehicles in the last decade.The sensor was mounted rigidly with the same stereo camera set and IMUas in the Hokuyo configuration. The Velodyne HDL-32e spins at 5 Hz andcollects range and intensity values from 32 lasers, thus sampling a 360degree field of view. For both configurations, the raw LIDAR rangevalues were integrated into a global coordinate frame using real-timepose estimates obtained from an extended Kalman filter that integratesIMU measurements with visual odometry. To aid in drift correction,incoming 3D point clouds were aligned to previously accumulated patchesvia real-time ICP (iterative closest point), thus feeding back driftcorrections to the 6DOF vision-based localization. To evaluate thesystem, data was collected from several outdoor environments with a mixof trees, buildings, vehicles, etc. The data sequences are 5 to 20minutes long and contain up to 0.5 billion points. The data wasprocessed in packets of 3000 points at a time and the compressionstatistics and performance times were aggregated across all sequencesfor the same sensor. The streaming framework was implemented in C++ anduses an octree class derived from the base class in Point Cloud Library.Tests were performed on a laptop with an i7 processor. The compressionrate was evaluated at a high (lossless) and low (lossy) resolution. TheHokuyo sensor was tested with 1 cm and 4 cm voxel size, and the Velodynesensor was tested with 4 cm and 8 cm voxel size. The normals arecomputed at the 4 cm voxel octree depth and using a search radius of 30cm with a maximum of 20 nearest neighbors. The normals were encoded with11 bits per voxel. Results are shown in FIGS. 18, 19 and 20 of theaforementioned U.S. Provisional Application Ser. No. 61/954,635.

In another configuration, a ground vehicle is equipped with multiplesensors including LIDAR, EO, IR (infrared) and multi-spectral sensors.These sensors are used to construct multi-modal 3D maps of theenvironment. A navigation module on the vehicle precisely locates andorients the vehicle in geo-spatial coordinates using data from GPS, IMU,odometer, compass and camera based sensors. The 3D pose estimated by thenavigation module is used to stitch together the LIDAR data to generate3D map data. 3D swath data can be further matched to improve the 3Dalignment. Final estimates of these poses allow highly accurate maplayer with all modalities of data co-located in geo-spatial coordinates.

A multi-modal change detection module (e.g., change detection module310) enables comparing the captured multi-modal 3D to previous collectsby aligning the two data sets automatically. Anomalous changes areautomatically detected and flagged for future examination by an analyst.Current environment conditions are used to infer if the change is due toenvironmental conditions e.g. different temperatures or naturalillumination at the two collects or if it is truly a real change basedon the business rules of the application.

In another configuration, an aerial vehicle is equipped with multiplesensors including LIDAR, EO, and IR sensors. These sensors are used toconstruct multi-modal 3D maps of the environment. A navigation module onthe aerial vehicle precisely locates and orients the vehicle ingeo-spatial coordinates using data from GPS, IMU, air-data, compass andcamera based sensors. The 3D pose estimated by the navigation module isused to stitch together the LIDAR data with the EO, and IR data andgeo-spatial reference terrain data of the earth. A multi-modal changedetection module enables comparing the captured multi-modal 3D toprevious collects by aligning the two data sets automatically. Anomalouschanges are automatically detected and flagged for future examination byan analyst.

For some applications, multi-sensor data collected from the air andground platforms can be combined to build an integrated multi-modal 3Dmap of the environment. The combination is typically be done by aligningthe two data sets. The alignment may be done at the pixel or voxel levelby matching 2D and 3D point clouds collected from the air and ground.Alternatively, the alignment can be done at the semantic level, wherehigh level features and objects such as roads, buildings, poles etc. aredetected in each of the ground and aerial collects and then theirdetected locations are used to match the two data sets. Analytics andchange detection can be performed on either the combined or separateaerial and ground collects and then fused together.

Implementation Examples

Referring now to FIG. 18, an exemplary implementation 1800 of thecomputing system 100 utilizes a commercial off-the-shelf (COTS)multi-processing computing platform. As described above, scalablemethods for processing large amounts of data for performing 3D modelingand 3D object classification using LIDAR and EO data acquired by bothaerial and ground sensors can be used. In particular, terabytes ofaerial LIDAR data can be processed using an off-the-shelf dual-quad coredesktop computer for performing LIDAR-based 3D object classification(ground extraction, vegetation clutter removal and 3D buildingsegmentation), automatic 2D-3D registration for texturing 3D models, forgeospatial areas in the range of thousands of square kilometers. Thescalability of the object recognition is provided by partitioning theinput LIDAR/EO data into tiles, which can be individually processedacross multiple CPUs as shown in FIG. 18. This data levelparallelization ensures a high degree of scalability of the processingwith minimal scripting language required (e.g., bash under Cygwin in thecase of the Windows operating system).

The illustrative system architecture of the implementation 1800 includesa multiprocessor task distribution manager 1810, which handles thedistribution of the sensor data from the various sensors (e.g., aerialLIDAR, ground LIDAR, 2D imagery) to processing modules 1812 (dataingestion and octree representation creation) and 1814(large-scale/proto-object extraction), distributes volumes of interestacross multiple CPUs (e.g., processor 1, processor N, where N is apositive integer), and handles large scale contextual identification1832. Each of the processors 1, N, can handle the object recognition andclassification tasks, e.g., proto-object extraction 1818, 1824;invariant feature computation 1820, 1826, proto-object unary andcontextual identification 1822, 1828). The multiprocessor taskdistribution manager 1810 also oversees the output of identificationresults and map representations to the visualization interface 1830 andaccesses and updates to/from the data stores including a featuredatabase, classifier parameters, and/or learned context data.

The task distribution manager 1810 has the role of communicating withindividual algorithmic components and to assign tasks to each of theprocessors using, e.g., the Message Passing Interface (MPI) protocolavailable both on Windows and Unix platforms. The system 1800 can use anoctree data structure to ensure an efficient memory usage over largeterrain areas and to access data at different resolution levels. Afterground removal, buildings and proto-objects are extracted at a coarsescale and each volume of interest (VOI) is sent to be further processedon multiple processors (e.g., processors 1 to N). Within each processor,proto-objects are extracted at finer resolution, and invariant featuresin 3D and 2D are computed for classification. The feature computationand the classification are the most computationally intensive modulesand offer a high degree of data level parallelization. In the laststage, the object classification results are collected from all thenodes to find large semantic structures using the bag-of-objectsapproach.

Referring now to FIG. 19, a simplified block diagram of an embodiment1900 of the real-time multi-modal 3D geospatial mapping, objectrecognition, scene annotation, analytics, and visualization computingsystem 100 is shown. While the illustrative embodiment 1900 is shown asinvolving multiple components and devices, it should be understood thatthe computing system 100 may constitute a single computing device, aloneor in combination with other devices. The embodiment 1900 includes asensor computing device 1910, which embodies features and functionalityof the multi-sensor data collection, analysis, recognition, andvisualization computing system 100 (e.g., the sensors 102, 104, 106 andthe platform 120), shown in FIG. 1, a data aggregation computing device1970, which embodies features and functionality of the multi-platformdata aggregation subsystem 148 shown in FIG. 1, and a user computingdevice 1950, which embodies features and functionality of the crowdsourcing app 152 shown in FIG. 1. The embodiment 1900 includes multipleadditional sensor computing devices 1940, 1942, each of which may beembodied in a similar manner as the sensor computing device 1910. Eachor any of the computing devices 1910, 1940, 1942, 1950, 1970 may be incommunication with one another via one or more networks or other typesof communication links, 1960.

The computing system 1900 or portions thereof may be distributed acrossmultiple computing devices that are connected to the network(s) 1960 asshown. In other embodiments, however, the computing system 1900 may belocated entirely on, for example, the sensor computing device 1910 orone of the devices 1950, 1970. In some embodiments, portions of thesystem 1900 may be incorporated into other systems or computerapplications (e.g. as a plugin). Such applications or systems mayinclude, for example, operating system software or GIS applications. Asused herein, “application” or “computer application” may refer to, amongother things, any type of computer program or group of computerprograms, whether implemented in software, hardware, or a combinationthereof, and includes self-contained, vertical, and/or shrink-wrappedsoftware applications, distributed and cloud-based applications, and/orothers. Portions of a computer application may be embodied as firmware,as one or more components of an operating system, a runtime library, anapplication programming interface (API), as a self-contained softwareapplication, or as a component of another software application, forexample.

The illustrative sensor computing device 1910 includes at least oneprocessor 1912 (e.g. a microprocessor, microcontroller, digital signalprocessor, etc.), memory 1914, and an input/output (I/O) subsystem 1916.The sensor computing device 1910 may be embodied as any type ofcomputing device capable of performing the functions described herein,such as a mobile robot, a navigation-capable electronic device, anunmanned or unpiloted aerial or ground vehicle, a personal computer(e.g.,. desktop, laptop, tablet, smart phone, body-mounted device,wearable device, etc.), a smart appliance, a server, an enterprisecomputer system, a network of computers, a combination of computers andother electronic devices, or other electronic devices. Although notspecifically shown, it should be understood that the I/O subsystem 1916typically includes, among other things, an I/O controller, a memorycontroller, and one or more I/O ports. The processor 1912 and the I/Osubsystem 1916 are communicatively coupled to the memory 1914. Thememory 1914 may be embodied as any type of suitable computer memorydevice (e.g., volatile memory such as various forms of random accessmemory).

The I/O subsystem 1916 is communicatively coupled to a number ofhardware and/or software components, including the components of thesystem 100 shown in FIGS. 1-6 and 7A, a data storage device 1918, anumber of sensors 1930, a user interface (UI) subsystem 1932, and acommunication subsystem 1934. The sensors 1930 may include one or morecameras (e.g., 2D and/or 3D still and/or video cameras), as well asother types of sensors such as IMUs, accelerometers, gyroscopes, GPSreceivers, LIDAR systems, and/or others. As used herein, a “camera” mayrefer to any type of 2D or 3D image sensor or other device that iscapable of acquiring and recording two-dimensional (2D) orthree-dimensional (3D) video images of portions of the real-worldenvironment, and may include cameras with one or more fixed cameraparameters and/or cameras having one or more variable parameters,fixed-location cameras (such as “stand-off” cameras that are installedin walls or ceilings), and/or mobile cameras (such as cameras that areintegrated with consumer electronic devices, such as laptop computers,smart phones, tablet computers, wearable electronic devices and/orothers.

The user interface subsystem 1932 includes one or more user inputdevices (e.g., a touchscreen, keyboard, virtual keypad, microphone,etc.) and one or more output devices (e.g., speakers, display devices,LEDs, etc.). The user interface subsystem 1932 may include devices suchas a touchscreen display, a touch-sensitive keypad, a kinetic sensorand/or other gesture-detecting device, an eye-tracking sensor, and/orother devices that are capable of detecting human interactions with acomputing device.

The devices 1950, 1970 are illustrated in FIG. 7 as being incommunication with the sensor computing device 1910 and/or other devices1940, 1942 by the network/communication links 1960. It should beunderstood that any or all of the devices 1940, 1942, 1950, 1970 may beintegrated with the sensor computing device 1910 or embodied as aseparate component.

The I/O subsystem 1916 is communicatively coupled to components of themulti-sensor data collection, analysis, recognition, and visualizationplatform 120 described above, one or more data storage devices (e.g.,machine readable storage media) 1918, a user interface (UI) subsystem1932, and a communication subsystem 1934. The storage media 1918 mayinclude one or more hard drives or other suitable data storage devices(e.g., flash memory, memory cards, memory sticks, and/or others). Insome embodiments, portions of the computing system 1900, e.g., theplatform 120, and/or other data and components, reside at leasttemporarily in the storage media 1918. Portions of the computing system1900, e.g., portions of the platform 120, may be copied to the memory1914 during operation of the computing device 1910, for fasterprocessing or other reasons.

The communication subsystem 1934 communicatively couples the sensorcomputing device 1910 to one or more other devices, systems, orcommunication networks, e.g., a local area network, wide area network,personal cloud, enterprise cloud, public cloud, and/or the Internet, forexample. Accordingly, the communication subsystem 1934 may include oneor more wired or wireless network interface software, firmware, orhardware, for example, as may be needed pursuant to the specificationsand/or design of the particular embodiment of the system 1900.

The user computing device 1950 and the data aggregation computing device1970 each may be embodied as any suitable type of computing device orpersonal electronic device capable of performing the functions describedherein, such as any of the aforementioned types of devices or otherelectronic devices. The illustrative user computing device 1950 and dataaggregation computing device 1970 each include components having thesame or similar names to components of the sensor computing device 1910,described above, and accordingly, those components of the computingdevices 1950, 1970 may be embodied similarly. Further, each of thedevices 1910, 1940, 1942, 1950, 1970 may include components similar tothose described above, and the computing system 1900 may include othercomponents, sub-components, and devices not illustrated in FIG. 19. Ingeneral, the components of the computing system 1900 are communicativelycoupled as shown in FIG. 19 by signal paths, which may be embodied asany type of wired or wireless signal paths capable of facilitatingcommunication between the respective devices and components.

Additional Examples

Illustrative examples of the technologies disclosed herein are providedbelow. An embodiment of the technologies may include any one or more,and any combination of, the examples described below.

In an example 1, a navigation-capable vehicle includes one or moreprocessors, and, in communication with the one or more processors: atwo-dimensional image sensor; a three-dimensional image sensor; one ormore sensors to determine motion, location, and orientation of thenavigation-capable vehicle; and one or more non-transitory machineaccessible storage media including instructions to cause thenavigation-capable vehicle to: temporally and spatially align sensordata received from the two-dimensional sensor, the three-dimensionalsensor, and the one or more motion, location, and orientation sensors;generate a map representation of a real world environment in a frame ofreference of the navigation-capable vehicle based on the temporally andspatially aligned sensor data; recognize a plurality of visual featuresin the map representation using one or more computer vision algorithms;and annotate one or more of the visual features in accordance withdomain-specific business logic.

An example 2 includes the subject matter of example 1, includinginstructions to cause the navigation-capable vehicle to present avisualization of the annotated visual features on the navigation-capablevehicle. An example 3 includes the subject matter of example 1 orexample 2, including instructions to cause the navigation-capablevehicle to estimate a navigation path for the navigation-capablevehicle. An example 4 includes the subject matter of any of examples1-3, including instructions to cause the navigation-capable vehicle toestimate a navigation path for the navigation-capable vehicle. Anexample 5 includes the subject matter of any of examples 1-4, includinginstructions to cause the navigation-capable vehicle to algorithmicallydetect changes in the visual features of the map representation overtime. An example 6 includes the subject matter of example 5, includinginstructions to cause the navigation-capable vehicle to transmit dataindicative of the detected changes to a ground-based computing device inreal time. An example 7 includes the subject matter of example 5 orexample 6, wherein the domain-specific business logic is to cause thenavigation-capable vehicle to detect a change in a visual feature of aphysical component of a ground-based utility service. An example 8includes the subject matter of any of examples 5-7, wherein thedomain-specific business logic is to cause the navigation-capablevehicle to detect a change in a visual feature of a natural resource. Anexample 9 includes the subject matter of any of examples 5-8, whereinthe domain-specific business logic is to cause the navigation-capablevehicle to detect a change in a visual feature of a man-made physicalstructure. An example 10 includes the subject matter of any of examples1-9, wherein the navigation-capable vehicle includes an unmanned aerialvehicle.

In an example 11, a multi-sensor data collection, analysis, recognition,and visualization platform includes instructions embodied in one or morenon-transitory computer readable storage media and executable by one ormore processors to cause a navigation-capable vehicle to: receive sensordata from a plurality of sensors including a two dimensional imagesensor, a three-dimensional image sensor, and one or more sensors todetermine motion, location, and orientation of the navigation-capablevehicle; temporally and spatially align the sensor data received fromthe two-dimensional sensor, the three-dimensional sensor, and the one ormore motion, location, and orientation sensors; generate a maprepresentation of the real world surroundings of the navigation-capablevehicle based on the temporally and spatially aligned sensor data;recognize a plurality of visual features in the map representation byexecuting one or more computer vision algorithms; annotate one or moreof the visual features in accordance with domain-specific businesslogic; and present a visualization of the annotated visual features onthe navigation-capable vehicle.

An example 12 includes the subject matter of example 11, includinginstructions to cause the navigation-capable vehicle to present thevisualization of the annotated visual features on the navigation-capablevehicle in real time. An example 13 includes the subject matter ofexample 11 or example 12, including instructions to cause thenavigation-capable vehicle to transmit the visualization of theannotated visual features to a ground-based computing device in realtime. An example 14 includes the subject matter of any of examples11-13, wherein the domain-specific business logic includes a changedetection algorithm to detect one or more domain-specific changes in thevisual features over time, and the platform includes instructions toannotate the visual features to identify the detected domain-specificchanges on the visualization. An example 15 includes the subject matterof any of examples 11-14, wherein the domain-specific business logicincludes an anomaly detection algorithm to detect one or moredomain-specific anomalies in the visual features over time, and theplatform includes instructions to annotate the visual features toidentify the detected domain-specific anomalies on the visualization.

In an example 16, a system for multi-sensor data collection, analysis,recognition, and visualization by a navigation-capable vehicle, thesystem including one or more computing devices configured to: temporallyand spatially align data received from a two-dimensional sensor, athree-dimensional sensor, and one or more motion, location, andorientation sensors; generate a map representation of the real worldsurroundings of the navigation-capable vehicle based on the temporallyand spatially aligned sensor data; recognize a plurality of visualfeatures in the map representation by executing one or more computervision algorithms; estimate a navigation path for the navigation-capablevehicle; annotate one or more of the visual features in accordance withdomain-specific business logic; and present a visualization of theannotated visual features on the navigation-capable vehicle.

An example 17 includes the subject matter of example 16, configured tointeractively modify the visualization of the annotated visual featuresin response to user input. An example 18 includes the subject matter ofexample 16 or example 17, configured to tag one or more of the annotatedvisual features in the visualization in response to user input. Anexample 19 includes the subject matter of any of examples 16-18,configured to execute domain-specific anomaly detection logic on thevisual features and annotate the visualization based on the execution ofthe domain-specific anomaly detection logic. An example 20 includes thesubject matter of any of examples 16-19, configured to executedomain-specific change detection logic on the visual features andannotate the visualization based on the execution of the domain-specificchange detection logic.

In an example 21, a mobile computing device includes one or moreprocessors, and, in communication with the one or more processors: oneor more image sensors configured to obtain two-dimensional image dataand three-dimensional image data; one or more non-transitory machineaccessible storage media including instructions to cause the mobilecomputing device to: temporally and spatially align the two-dimensionalimage data and three-dimensional image data; generate a maprepresentation of a geo-spatial area of the real world surroundings ofthe mobile computing device based on the temporally and spatiallyaligned two-dimensional and three-dimensional image data; and recognizea plurality of visual features in the map representation, using one ormore computer vision algorithms to: recognize larger-scale objects;recognize smaller-scale objects by iteratively performing context-freeobject identification and contextual object identification; andrecognize a complex object including a plurality of the smaller-scaleobjects, using a classifier.

An example 22 includes the subject matter of example 21, includinginstructions to cause the mobile computing device to detect thelarger-scale objects by determining a contextual frame of reference, anduse the contextual frame of reference to identify the larger-scaleobjects. An example 23 includes the subject matter of example 21 orexample 22, including instructions to cause the mobile computing deviceto recognize the larger-scale objects by executing an invariantthree-dimensional feature detection algorithm directly on point clouddata obtained from at least one of the image sensors. An example 24includes the subject matter of example 23, including instructions tocause the mobile computing device to recognize the larger-scale objectsby executing an invariant two-dimensional feature detection algorithm.An example 25 includes the subject matter of any of examples 21-24,including instructions to cause the mobile computing device to recognizethe larger-scale objects by executing an invariant two-dimensionalfeature detection algorithm. An example 26 includes the subject matterof any of examples 21-25, including instructions to cause the mobilecomputing device to recognize the smaller-scale objects by executing acontext-free feature-sharing algorithm. An example 27 includes thesubject matter of example 26, including instructions to cause the mobilecomputing device to recognize the smaller-scale objects by obtainingcontext information and classifying the smaller-scale objects based onthe context information. An example 28 includes the subject matter ofexample 26 or example 27, including instructions to cause the mobilecomputing device to recognize the complex objects by executing acontextual bag of objects algorithm.

In an example 29, an object/scene recognition system includesinstructions embodied in one or more non-transitory computer readablestorage media to and executable by one or more processors to cause amobile computing device to: obtain two-dimensional image data andthree-dimensional image data from one or more image sensors; temporallyand spatially align the two-dimensional image data and three-dimensionalimage data; recognize a plurality of visual features in the image data,using one or more computer vision algorithms to: recognize larger-scaleobjects; recognize smaller-scale objects by iteratively performingcontext-free object identification and contextual object identification;and recognize a complex object including a plurality of the recognizedsmaller-scale objects, using a classifier.

An example 30 includes the subject matter of example 29, includinginstructions to cause the mobile computing device to detect thelarger-scale objects by determining a contextual frame of reference anduse the contextual frame of reference to identify the larger-scaleobjects. An example 31 includes the subject matter of example 29 orexample 30, including instructions to cause the mobile computing deviceto recognize the larger-scale objects by executing an invariantthree-dimensional feature detection algorithm directly on point clouddata obtained from at least one of the image sensors. An example 32includes the subject matter of example 31, including instructions tocause the mobile computing device to recognize the larger-scale objectsby executing an invariant two-dimensional feature detection algorithm.An example 33 includes the subject matter of any of examples 29-32,including instructions to cause the mobile computing device to recognizethe larger-scale objects by executing an invariant two-dimensionalfeature detection algorithm. An example 34 includes the subject matterof example 30, including instructions to cause the mobile computingdevice to recognize the smaller-scale objects by executing acontext-free feature-sharing algorithm. An example 35 includes thesubject matter of example 34, including instructions to cause the mobilecomputing device to recognize the smaller-scale objects by obtainingcontext information and classifying the smaller-scale objects based onthe context information. An example 36 includes the subject matter ofexample 34 or example 35, including instructions to cause the mobilecomputing device to recognize the complex objects by executing acontextual bag of objects algorithm.

In an example 37, an object/scene recognition method includes, with oneor more mobile computing devices: obtaining two-dimensional image dataand three-dimensional image data from one or more image sensors;temporally and spatially aligning the two-dimensional image data and thethree-dimensional image data; and recognizing a plurality of visualfeatures in the image data by: recognizing larger-scale objects;recognizing smaller-scale objects by iteratively performing context-freeobject identification and contextual object identification; andrecognizing a complex object including a plurality of the smaller-scaleobjects using a classifier.

An example 38 includes the subject matter of example 37, includingdetermining a contextual frame of reference and using the contextualframe of reference to identify the larger-scale objects. An example 39includes the subject matter of example 37 or example 38, includingexecuting an invariant three-dimensional feature detection algorithmdirectly on point cloud data obtained from at least one of the imagesensors. An example 40 includes the subject matter of example 38,including, iteratively: executing a context-free feature-sharingalgorithm to recognize the smaller-scale objects, obtaining contextinformation, and classifying the smaller-scale objects based on thecontext information.

General Considerations

In the foregoing description, numerous specific details, examples, andscenarios are set forth in order to provide a more thoroughunderstanding of the present disclosure. It will be appreciated,however, that embodiments of the disclosure may be practiced withoutsuch specific details. Further, such examples and scenarios are providedfor illustration, and are not intended to limit the disclosure in anyway. Those of ordinary skill in the art, with the included descriptions,should be able to implement appropriate functionality without undueexperimentation.

References in the specification to “an embodiment,” etc., indicate thatthe embodiment described may include a particular feature, structure, orcharacteristic, but every embodiment may not necessarily include theparticular feature, structure, or characteristic. Such phrases are notnecessarily referring to the same embodiment. Further, when a particularfeature, structure, or characteristic is described in connection with anembodiment, it is believed to be within the knowledge of one skilled inthe art to affect such feature, structure, or characteristic inconnection with other embodiments whether or not explicitly indicated.

Embodiments in accordance with the disclosure may be implemented inhardware, firmware, software, or any combination thereof. Embodimentsmay also be implemented as instructions stored using one or moremachine-readable media, which may be read and executed by one or moreprocessors. A machine-readable medium may include any mechanism forstoring or transmitting information in a form readable by a machine(e.g., a computing device or a “virtual machine” running on one or morecomputing devices). For example, a machine-readable medium may includeany suitable form of volatile or non-volatile memory.

Modules, data structures, blocks, and the like are referred to as suchfor ease of discussion, and are not intended to imply that any specificimplementation details are required. For example, any of the describedmodules and/or data structures may be combined or divided intosub-modules, sub-processes or other units of computer code or data asmay be required by a particular design or implementation. In thedrawings, specific arrangements or orderings of schematic elements maybe shown for ease of description. However, the specific ordering orarrangement of such elements is not meant to imply that a particularorder or sequence of processing, or separation of processes, is requiredin all embodiments. In general, schematic elements used to representinstruction blocks or modules may be implemented using any suitable formof machine-readable instruction, and each such instruction may beimplemented using any suitable programming language, library,application-programming interface (API), and/or other softwaredevelopment tools or frameworks. Similarly, schematic elements used torepresent data or information may be implemented using any suitableelectronic arrangement or data structure. Further, some connections,relationships or associations between elements may be simplified or notshown in the drawings so as not to obscure the disclosure. Thisdisclosure is to be considered as exemplary and not restrictive incharacter, and all changes and modifications that come within the spiritof the disclosure are desired to be protected.

The invention claimed is:
 1. A mobile computing device, comprising: oneor more processors, and, in communication with the one or moreprocessors: one or more image sensors configured to obtaintwo-dimensional image data and three-dimensional image data; one or morenon-transitory machine accessible storage media comprising instructionsto cause the mobile computing device to: temporally and spatially alignthe two-dimensional image data and three-dimensional image data;generate a map representation of a geo-spatial area of the real worldsurroundings of the mobile computing device based on the temporally andspatially aligned two-dimensional and three-dimensional image data; andrecognize a plurality of visual features in the map representation,using one or more computer vision algorithms to: recognize larger-scaleobjects; recognize smaller-scale objects by iteratively performingcontext-free object identification and contextual object identification;and recognize a complex object comprising a plurality of thesmaller-scale objects, using a classifier.
 2. The mobile computingdevice of claim 1, comprising instructions to cause the mobile computingdevice to detect the larger-scale objects by determining a contextualframe of reference, and use the contextual frame of reference toidentify the larger-scale objects.
 3. The mobile computing device ofclaim 1, comprising instructions to cause the mobile computing device torecognize the larger-scale objects by executing an invariantthree-dimensional feature detection algorithm directly on point clouddata obtained from at least one of the image sensors.
 4. The mobilecomputing device of claim 3, comprising instructions to cause the mobilecomputing device to recognize the larger-scale objects by executing aninvariant two-dimensional feature detection algorithm.
 5. The mobilecomputing device of claim 1, comprising instructions to cause the mobilecomputing device to recognize the larger-scale objects by executing aninvariant two-dimensional feature detection algorithm.
 6. The mobilecomputing device of claim 2, comprising instructions to cause the mobilecomputing device to recognize the smaller-scale objects by executing acontext-free feature-sharing algorithm.
 7. The mobile computing deviceof claim 6, comprising instructions to cause the mobile computing deviceto recognize the smaller-scale objects by obtaining context informationand classifying the smaller-scale objects based on the contextinformation.
 8. The mobile computing device of claim 6, comprisinginstructions to cause the mobile computing device to recognize thecomplex objects by executing a contextual bag of objects algorithm. 9.An object/scene recognition system comprising instructions embodied inone or more non-transitory computer readable storage media to andexecutable by one or more processors to cause a mobile computing deviceto: obtain two-dimensional image data and three-dimensional image datafrom one or more image sensors; temporally and spatially align thetwo-dimensional image data and three-dimensional image data; recognize aplurality of visual features in the image data, using one or morecomputer vision algorithms to: recognize larger-scale objects; recognizesmaller-scale objects by iteratively performing context-free objectidentification and contextual object identification; and recognize acomplex object comprising a plurality of the recognized smaller-scaleobjects, using a classifier.
 10. The system of claim 9, comprisinginstructions to cause the mobile computing device to detect thelarger-scale objects by determining a contextual frame of reference anduse the contextual frame of reference to identify the larger-scaleobjects.
 11. The system of claim 9, comprising instructions to cause themobile computing device to recognize the larger-scale objects byexecuting an invariant three-dimensional feature detection algorithmdirectly on point cloud data obtained from at least one of the imagesensors.
 12. The system of claim 11, comprising instructions to causethe mobile computing device to recognize the larger-scale objects byexecuting an invariant two-dimensional feature detection algorithm. 13.The system of claim 9, comprising instructions to cause the mobilecomputing device to recognize the larger-scale objects by executing aninvariant two-dimensional feature detection algorithm.
 14. The system ofclaim 10, comprising instructions to cause the mobile computing deviceto recognize the smaller-scale objects by executing a context-freefeature-sharing algorithm.
 15. The mobile computing device of claim 14,comprising instructions to cause the mobile computing device torecognize the smaller-scale objects by obtaining context information andclassifying the smaller-scale objects based on the context information.16. The mobile computing device of claim 14, comprising instructions tocause the mobile computing device to recognize the complex objects byexecuting a contextual bag of objects algorithm.
 17. An object/scenerecognition method comprising, with one or more mobile computingdevices: obtaining two-dimensional image data and three-dimensionalimage data from one or more image sensors; temporally and spatiallyaligning the two-dimensional image data and the three-dimensional imagedata; and recognizing a plurality of visual features in the image databy: recognizing larger-scale objects; recognizing smaller-scale objectsby iteratively performing context-free object identification andcontextual object identification; and recognizing a complex objectcomprising a plurality of the smaller-scale objects using a classifier.18. The method of claim 17, comprising determining a contextual frame ofreference and using the contextual frame of reference to identify thelarger-scale objects.
 19. The method of claim 17, comprising executingan invariant three-dimensional feature detection algorithm directly onpoint cloud data obtained from at least one of the image sensors. 20.The method of claim 18, comprising, iteratively: executing acontext-free feature-sharing algorithm to recognize the smaller-scaleobjects, obtaining context information, and classifying thesmaller-scale objects based on the context information.